Login
[x]
Log in using an account from:
Fedora Account System
Red Hat Associate
Red Hat Customer
Or login using a Red Hat Bugzilla account
Forgot Password
Login:
Hide Forgot
Create an Account
Red Hat Bugzilla – Attachment 609123 Details for
Bug 853747
[Log][engine] NullPointerException in spmStart in case storage is inaccessible or not connected (Command SpmStartVDS execution failed. Exception: NullPointerException)
[?]
New
Simple Search
Advanced Search
My Links
Browse
Requests
Reports
Current State
Search
Tabular reports
Graphical reports
Duplicates
Other Reports
User Changes
Plotly Reports
Bug Status
Bug Severity
Non-Defaults
|
Product Dashboard
Help
Page Help!
Bug Writing Guidelines
What's new
Browser Support Policy
5.0.4.rh83 Release notes
FAQ
Guides index
User guide
Web Services
Contact
Legal
This site requires JavaScript to be enabled to function correctly, please enable it.
engine logs
engine.log (text/x-log), 1.65 MB, created by
Gadi Ickowicz
on 2012-09-02 15:36:13 UTC
(
hide
)
Description:
engine logs
Filename:
MIME Type:
Creator:
Gadi Ickowicz
Created:
2012-09-02 15:36:13 UTC
Size:
1.65 MB
patch
obsolete
>2012-09-02 12:03:38,867 INFO [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-2) Start time: 9/2/12 12:03 PM >2012-09-02 12:03:39,294 WARN [org.ovirt.engine.core.utils.ConfigUtilsBase] (MSC service thread 1-2) Could not find enum value for option: CbcCheckOnVdsChange >2012-09-02 12:03:39,323 WARN [org.ovirt.engine.core.utils.ConfigUtilsBase] (MSC service thread 1-2) Could not find enum value for option: CAEngineKey >2012-09-02 12:03:39,361 WARN [org.ovirt.engine.core.utils.ConfigUtilsBase] (MSC service thread 1-2) Could not find enum value for option: MinimalETLVersion >2012-09-02 12:03:40,135 WARN [org.ovirt.engine.core.utils.ConfigUtilsBase] (MSC service thread 1-2) Could not find enum value for option: ScriptsPath >2012-09-02 12:03:40,136 WARN [org.ovirt.engine.core.utils.ConfigUtilsBase] (MSC service thread 1-2) Could not find enum value for option: SQLServerI18NPrefix >2012-09-02 12:03:40,323 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (MSC service thread 1-2) Failed to decrypt Data must start with zero >2012-09-02 12:03:40,323 ERROR [org.ovirt.engine.core.dal.dbbroker.generic.DBConfigUtils] (MSC service thread 1-2) Failed to decrypt value for property TruststorePass will be used encrypted value >2012-09-02 12:03:40,370 WARN [org.ovirt.engine.core.utils.ConfigUtilsBase] (MSC service thread 1-2) Could not find enum value for option: ENGINEEARLib >2012-09-02 12:03:40,512 INFO [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-2) VDSBrokerFrontend: 9/2/12 12:03 PM >2012-09-02 12:03:40,524 INFO [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-2) CpuFlagsManager: 9/2/12 12:03 PM >2012-09-02 12:03:40,525 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-2) Setting audit clean up manager to run at: 35 35 3 * * ? >2012-09-02 12:03:40,580 INFO [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-2) AuditLogCleanupManager: 9/2/12 12:03 PM >2012-09-02 12:03:40,589 INFO [org.ovirt.engine.core.bll.TagsDirector] (MSC service thread 1-2) TagsDirector initialization >2012-09-02 12:03:40,674 INFO [org.ovirt.engine.core.bll.TagsDirector] (MSC service thread 1-2) Tag root added to tree >2012-09-02 12:03:40,727 INFO [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-2) TagsDirector: 9/2/12 12:03 PM >2012-09-02 12:03:40,740 INFO [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-2) IsoDomainListSyncronizer: 9/2/12 12:03 PM >2012-09-02 12:03:41,352 INFO [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-2) InitHandlers: 9/2/12 12:03 PM >2012-09-02 12:03:41,376 WARN [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-2) Code VDS_FENCING_OPERATION_FAILED appears more then once in string table. >2012-09-02 12:03:41,377 INFO [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-2) ErrorTranslator: 9/2/12 12:03 PM >2012-09-02 12:03:41,385 INFO [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-2) VdsErrorTranslator: 9/2/12 12:03 PM >2012-09-02 12:03:41,386 INFO [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-2) Mark uncompleted jobs as UNKNOWN: 9/2/12 12:03 PM >2012-09-02 12:03:41,483 INFO [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-2) JobRepositoryCleanupManager: 9/2/12 12:03 PM >2012-09-02 12:03:41,487 INFO [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-2) AutoRecoveryManager: 9/2/12 12:03 PM >2012-09-02 12:03:41,495 INFO [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-2) ExecutionMessageDirector: 9/2/12 12:03 PM >2012-09-02 12:03:41,577 INFO [org.ovirt.engine.core.bll.adbroker.UsersDomainsCacheManagerService] (MSC service thread 1-2) UsersDomainsCacheManager: 9/2/12 12:03 PM >2012-09-02 12:03:41,612 INFO [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (MSC service thread 1-1) InitResourceManager: 9/2/12 12:03 PM >2012-09-02 12:03:41,628 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (MSC service thread 1-1) ResourceManager::ResourceManager::Entered >2012-09-02 12:03:41,810 INFO [org.ovirt.engine.core.bll.adbroker.UsersDomainsCacheManagerService] (MSC service thread 1-2) DbUserCacheManager: 9/2/12 12:03 PM >2012-09-02 12:03:41,940 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-1) DbUserCacheManager::refreshAllUserData() - entered >2012-09-02 12:03:42,258 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (MSC service thread 1-1) AsyncTaskManager: Initialization of AsyncTaskManager completed successfully. >2012-09-02 12:03:42,259 INFO [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (MSC service thread 1-1) AsyncTaskManager: 9/2/12 12:03 PM >2012-09-02 12:03:42,263 INFO [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (MSC service thread 1-1) VdsLoadBalancer: 9/2/12 12:03 PM >2012-09-02 12:03:42,276 INFO [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (pool-4-thread-1) MacPoolManager started: 9/2/12 12:03 PM >2012-09-02 12:03:42,343 INFO [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (pool-4-thread-1) MacPoolManager finished: 9/2/12 12:03 PM >2012-09-02 12:05:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-19) Autorecovering hosts is disabled, skipping >2012-09-02 12:05:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-19) Autorecovering storage domains is disabled, skipping >2012-09-02 12:10:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-97) Autorecovering hosts is disabled, skipping >2012-09-02 12:10:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-97) Autorecovering storage domains is disabled, skipping >2012-09-02 12:15:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-75) Autorecovering hosts is disabled, skipping >2012-09-02 12:15:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-75) Autorecovering storage domains is disabled, skipping >2012-09-02 12:20:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-52) Autorecovering hosts is disabled, skipping >2012-09-02 12:20:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-52) Autorecovering storage domains is disabled, skipping >2012-09-02 12:25:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-30) Autorecovering hosts is disabled, skipping >2012-09-02 12:25:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-30) Autorecovering storage domains is disabled, skipping >2012-09-02 12:30:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-7) Autorecovering hosts is disabled, skipping >2012-09-02 12:30:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-7) Autorecovering storage domains is disabled, skipping >2012-09-02 12:35:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-86) Autorecovering hosts is disabled, skipping >2012-09-02 12:35:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-86) Autorecovering storage domains is disabled, skipping >2012-09-02 12:40:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-63) Autorecovering hosts is disabled, skipping >2012-09-02 12:40:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-63) Autorecovering storage domains is disabled, skipping >2012-09-02 12:45:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-41) Autorecovering hosts is disabled, skipping >2012-09-02 12:45:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-41) Autorecovering storage domains is disabled, skipping >2012-09-02 12:50:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-18) Autorecovering hosts is disabled, skipping >2012-09-02 12:50:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-18) Autorecovering storage domains is disabled, skipping >2012-09-02 12:55:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-96) Autorecovering hosts is disabled, skipping >2012-09-02 12:55:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-96) Autorecovering storage domains is disabled, skipping >2012-09-02 13:00:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-73) Autorecovering hosts is disabled, skipping >2012-09-02 13:00:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-73) Autorecovering storage domains is disabled, skipping >2012-09-02 13:03:41,960 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-30) DbUserCacheManager::refreshAllUserData() - entered >2012-09-02 13:05:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-53) Autorecovering hosts is disabled, skipping >2012-09-02 13:05:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-53) Autorecovering storage domains is disabled, skipping >2012-09-02 13:10:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-31) Autorecovering hosts is disabled, skipping >2012-09-02 13:10:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-31) Autorecovering storage domains is disabled, skipping >2012-09-02 13:15:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-9) Autorecovering hosts is disabled, skipping >2012-09-02 13:15:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-9) Autorecovering storage domains is disabled, skipping >2012-09-02 13:20:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-86) Autorecovering hosts is disabled, skipping >2012-09-02 13:20:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-86) Autorecovering storage domains is disabled, skipping >2012-09-02 13:25:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-63) Autorecovering hosts is disabled, skipping >2012-09-02 13:25:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-63) Autorecovering storage domains is disabled, skipping >2012-09-02 13:30:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-41) Autorecovering hosts is disabled, skipping >2012-09-02 13:30:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-41) Autorecovering storage domains is disabled, skipping >2012-09-02 13:32:38,077 INFO [org.ovirt.engine.core.bll.LoginAdminUserCommand] (ajp-/127.0.0.1:8009-5) Checking if user admin@internal is an admin, result true >2012-09-02 13:32:38,080 INFO [org.ovirt.engine.core.bll.LoginAdminUserCommand] (ajp-/127.0.0.1:8009-5) Running command: LoginAdminUserCommand internal: false. >2012-09-02 13:32:38,145 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8009-5) AuditLogType: UNASSIGNED not exist in string table >2012-09-02 13:32:38,145 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8009-5) AuditLogType: VDS_NETWORK_MTU_DIFFER_FROM_LOGICAL_NETWORK not exist in string table >2012-09-02 13:32:38,146 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8009-5) AuditLogType: STORAGE_ACTIVATE_ASYNC not exist in string table >2012-09-02 13:32:38,146 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8009-5) AuditLogType: DWH_STOPPED not exist in string table >2012-09-02 13:32:38,146 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8009-5) AuditLogType: DWH_STARTED not exist in string table >2012-09-02 13:32:38,146 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8009-5) AuditLogType: DWH_ERROR not exist in string table >2012-09-02 13:32:38,156 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8009-5) AuditLogType: USER_ATTACH_DISK_TO_VM not have severity. Assumed Normal >2012-09-02 13:32:38,156 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8009-5) AuditLogType: USER_FAILED_ATTACH_DISK_TO_VM not have severity. Assumed Normal >2012-09-02 13:32:38,157 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8009-5) AuditLogType: USER_DETACH_DISK_FROM_VM not have severity. Assumed Normal >2012-09-02 13:32:38,157 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8009-5) AuditLogType: USER_FAILED_DETACH_DISK_FROM_VM not have severity. Assumed Normal >2012-09-02 13:32:38,157 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8009-5) AuditLogType: VDS_NETWORK_MTU_DIFFER_FROM_LOGICAL_NETWORK not have severity. Assumed Normal >2012-09-02 13:32:39,074 WARN [org.ovirt.engine.core.dal.dbbroker.generic.DBConfigUtils] (ajp-/127.0.0.1:8009-8) Adding new value to configuration cache. >2012-09-02 13:32:39,099 WARN [org.ovirt.engine.core.dal.dbbroker.generic.DBConfigUtils] (ajp-/127.0.0.1:8009-8) Didnt find the value of DBPagingType in DB for version general - using default: Offset >2012-09-02 13:32:39,101 WARN [org.ovirt.engine.core.dal.dbbroker.generic.DBConfigUtils] (ajp-/127.0.0.1:8009-8) Adding new value to configuration cache. >2012-09-02 13:32:39,101 WARN [org.ovirt.engine.core.dal.dbbroker.generic.DBConfigUtils] (ajp-/127.0.0.1:8009-8) Didnt find the value of DBPagingSyntax in DB for version general - using default: OFFSET (%1$s -1) LIMIT %2$s >2012-09-02 13:32:39,102 WARN [org.ovirt.engine.core.dal.dbbroker.generic.DBConfigUtils] (ajp-/127.0.0.1:8009-8) Adding new value to configuration cache. >2012-09-02 13:32:39,102 WARN [org.ovirt.engine.core.dal.dbbroker.generic.DBConfigUtils] (ajp-/127.0.0.1:8009-8) Didnt find the value of DBSearchTemplate in DB for version general - using default: SELECT * FROM (%2$s) %1$s) as T1 %3$s >2012-09-02 13:32:39,486 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8009-7) calling GetConfigurationValueQuery (CustomPublicConfig_AppsWebSite) with null version, using default general for version >2012-09-02 13:33:33,913 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8009-4) Checking if user admin@internal is an admin, result true >2012-09-02 13:33:33,913 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8009-4) Running command: LoginUserCommand internal: false. >2012-09-02 13:33:33,956 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8009-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2012-09-02 13:33:36,023 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8009-3) Checking if user admin@internal is an admin, result true >2012-09-02 13:33:36,024 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8009-3) Running command: LoginUserCommand internal: false. >2012-09-02 13:33:36,057 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8009-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2012-09-02 13:33:50,464 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8009-7) Checking if user admin@internal is an admin, result true >2012-09-02 13:33:50,465 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8009-7) Running command: LoginUserCommand internal: false. >2012-09-02 13:33:50,530 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8009-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2012-09-02 13:33:50,791 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8009-4) Checking if user admin@internal is an admin, result true >2012-09-02 13:33:50,791 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8009-4) Running command: LoginUserCommand internal: false. >2012-09-02 13:33:50,842 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8009-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2012-09-02 13:33:51,965 INFO [org.ovirt.engine.core.bll.storage.AddEmptyStoragePoolCommand] (ajp-/127.0.0.1:8009-1) [24cef2c] Running command: AddEmptyStoragePoolCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2012-09-02 13:33:53,800 INFO [org.ovirt.engine.core.bll.AddVdsGroupCommand] (ajp-/127.0.0.1:8009-7) [48932da2] Running command: AddVdsGroupCommand internal: false. Entities affected : ID: ba2bec5d-7b0c-4fbc-941f-a2a62457907a Type: StoragePool >2012-09-02 13:33:56,641 WARN [org.ovirt.engine.core.dal.dbbroker.generic.DBConfigUtils] (ajp-/127.0.0.1:8009-7) [145fe51d] Adding new value to configuration cache. >2012-09-02 13:33:56,641 WARN [org.ovirt.engine.core.dal.dbbroker.generic.DBConfigUtils] (ajp-/127.0.0.1:8009-7) [145fe51d] Didnt find the value of BootstrapNodeIDCommand in DB for version general - using default: dmidecode | awk ' /UUID/{ print $2; } ' | tr ' >' '_' && cat /sys/class/net/*/address | sed -e '/00:00:00:00/d' -e '/^$/d' | sort -u | head -n 1 >2012-09-02 13:33:56,641 INFO [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (ajp-/127.0.0.1:8009-7) SSH execute 10.35.102.10 'dmidecode | awk ' /UUID/{ print $2; } ' | tr ' >' '_' && cat /sys/class/net/*/address | sed -e '/00:00:00:00/d' -e '/^$/d' | sort -u | head -n 1' >2012-09-02 13:33:56,851 INFO [org.ovirt.engine.core.bll.AddVdsCommand] (ajp-/127.0.0.1:8009-7) [145fe51d] Running command: AddVdsCommand internal: false. Entities affected : ID: b1600626-f4e9-11e1-b9cc-001a4a169705 Type: VdsGroups >2012-09-02 13:33:57,077 INFO [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp-/127.0.0.1:8009-7) [5fcd56b3] Running command: AddVdsSpmIdCommand internal: true. Entities affected : ID: b32eacf0-f4e9-11e1-973c-001a4a169705 Type: VDS >2012-09-02 13:33:57,126 INFO [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp-/127.0.0.1:8009-7) [5fcd56b3] Lock freed to object EngineLock [exclusiveLocks= key: ba2bec5d-7b0c-4fbc-941f-a2a62457907a value: REGISTER_VDS >, sharedLocks= ] >2012-09-02 13:33:57,135 ERROR [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp-/127.0.0.1:8009-7) [5fcd56b3] Cannot get vdsManager for vdsid=b32eacf0-f4e9-11e1-973c-001a4a169705 >2012-09-02 13:33:57,135 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (ajp-/127.0.0.1:8009-7) [5fcd56b3] START, RemoveVdsVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705), log id: 49598f44 >2012-09-02 13:33:57,136 ERROR [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp-/127.0.0.1:8009-7) [5fcd56b3] Cannot get vdsManager for vdsid=b32eacf0-f4e9-11e1-973c-001a4a169705 >2012-09-02 13:33:57,136 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (ajp-/127.0.0.1:8009-7) [5fcd56b3] FINISH, RemoveVdsVDSCommand, log id: 49598f44 >2012-09-02 13:33:57,141 ERROR [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp-/127.0.0.1:8009-7) [5fcd56b3] Cannot get vdsManager for vdsid=b32eacf0-f4e9-11e1-973c-001a4a169705 >2012-09-02 13:33:57,141 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8009-7) [5fcd56b3] START, AddVdsVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705), log id: 23d3551f >2012-09-02 13:33:57,141 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8009-7) [5fcd56b3] AddVds - entered , starting logic to add VDS b32eacf0-f4e9-11e1-973c-001a4a169705 >2012-09-02 13:33:57,166 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8009-7) [5fcd56b3] AddVds - VDS b32eacf0-f4e9-11e1-973c-001a4a169705 was added, will try to add it to the resource manager >2012-09-02 13:33:57,180 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ajp-/127.0.0.1:8009-7) [5fcd56b3] Eneterd VdsManager:constructor >2012-09-02 13:33:57,186 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ajp-/127.0.0.1:8009-7) [5fcd56b3] vdsBroker(10.35.102.10,54,321) >2012-09-02 13:33:57,331 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp-/127.0.0.1:8009-7) [5fcd56b3] ResourceManager::AddVds - VDS b32eacf0-f4e9-11e1-973c-001a4a169705 was added to the Resource Manager >2012-09-02 13:33:57,331 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8009-7) [5fcd56b3] FINISH, AddVdsVDSCommand, log id: 23d3551f >2012-09-02 13:33:57,442 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-4-thread-7) [5ac74306] Running command: InstallVdsCommand internal: true. Entities affected : ID: b32eacf0-f4e9-11e1-973c-001a4a169705 Type: VDS >2012-09-02 13:33:57,449 WARN [org.ovirt.engine.core.dal.dbbroker.generic.DBConfigUtils] (pool-4-thread-7) [5ac74306] Adding new value to configuration cache. >2012-09-02 13:33:57,449 WARN [org.ovirt.engine.core.dal.dbbroker.generic.DBConfigUtils] (pool-4-thread-7) [5ac74306] Didnt find the value of BootstrapCommand in DB for version general - using default: umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar -C "${MYTMP}" --no-same-permissions -o -x && "${MYTMP}"/setup -c 'ssl={server_SSL_enabled};management_port={management_port}' -O '{OrganizationName}' -t {utc_time} {OverrideFirewall} -S {SSHKey} {EnginePort} -b {virt-placeholder} {gluster-placeholder} {URL1} {URL1} {vds-server} {GUID} {RunFlag} >2012-09-02 13:33:57,450 WARN [org.ovirt.engine.core.dal.dbbroker.generic.DBConfigUtils] (pool-4-thread-7) [5ac74306] Adding new value to configuration cache. >2012-09-02 13:33:57,450 WARN [org.ovirt.engine.core.dal.dbbroker.generic.DBConfigUtils] (pool-4-thread-7) [5ac74306] Didnt find the value of BootstrapPackageName in DB for version general - using default: vdsm-bootstrap-2.tar >2012-09-02 13:33:57,450 WARN [org.ovirt.engine.core.dal.dbbroker.generic.DBConfigUtils] (pool-4-thread-7) [5ac74306] Adding new value to configuration cache. >2012-09-02 13:33:57,450 WARN [org.ovirt.engine.core.dal.dbbroker.generic.DBConfigUtils] (pool-4-thread-7) [5ac74306] Didnt find the value of BootstrapPackageDirectory in DB for version general - using default: /usr/share/vdsm-bootstrap/interface-2 >2012-09-02 13:33:57,455 WARN [org.ovirt.engine.core.dal.dbbroker.generic.DBConfigUtils] (pool-4-thread-7) [5ac74306] Adding new value to configuration cache. >2012-09-02 13:33:57,455 WARN [org.ovirt.engine.core.dal.dbbroker.generic.DBConfigUtils] (pool-4-thread-7) [5ac74306] Didnt find the value of BootstrapCacheRefreshInterval in DB for version general - using default: 10,000 >2012-09-02 13:33:57,457 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-4-thread-7) [5ac74306] Before Installation pool-4-thread-7 >2012-09-02 13:33:57,457 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-7) [5ac74306] Installation of 10.35.102.10. Executing installation stage. (Stage: Starting Host installation) >2012-09-02 13:33:57,457 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-7) [5ac74306] Installation of 10.35.102.10. Executing installation stage. (Stage: Connecting to Host) >2012-09-02 13:33:57,562 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-7) [5ac74306] Installation of 10.35.102.10. Received message: <BSTRAP component='RHEV_INSTALL' status='OK' message='Connected to Host 10.35.102.10 with SSH key fingerprint: e8:fc:9d:1d:1a:d9:bb:3f:fb:35:cf:7e:01:54:20:28'/>. FYI. (Stage: Connecting to Host) >2012-09-02 13:33:57,778 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-7) [5ac74306] Installation of 10.35.102.10. Successfully connected to server ssh. (Stage: Connecting to Host) >2012-09-02 13:33:57,778 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-7) [5ac74306] Installation of 10.35.102.10. Executing installation stage. (Stage: Get the unique vds id) >2012-09-02 13:33:57,778 INFO [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-4-thread-7) SSH execute 10.35.102.10 'dmidecode | awk ' /UUID/{ print $2; } ' | tr ' >' '_' && cat /sys/class/net/*/address | sed -e '/00:00:00:00/d' -e '/^$/d' | sort -u | head -n 1' >2012-09-02 13:33:57,891 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-3) Installation of 10.35.102.10. Received message: AE0A6238-9CE0-3416-8C84-16A224E70650_00:14:5e:17:cf:e4. FYI. (Stage: Get the unique vds id) >2012-09-02 13:33:57,895 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-3) Installation of 10.35.102.10. Assigning unique id AE0A6238-9CE0-3416-8C84-16A224E70650_00:14:5e:17:cf:e4 to Host. (Stage: Get the unique vds id) >2012-09-02 13:33:57,919 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-7) [5ac74306] Installation of 10.35.102.10. Executing installation stage. (Stage: Upload Installation script to Host) >2012-09-02 13:33:57,920 WARN [org.ovirt.engine.core.dal.dbbroker.generic.DBConfigUtils] (pool-4-thread-7) [5ac74306] Adding new value to configuration cache. >2012-09-02 13:33:57,920 WARN [org.ovirt.engine.core.dal.dbbroker.generic.DBConfigUtils] (pool-4-thread-7) [5ac74306] Didnt find the value of SSHKeyAlias in DB for version general - using default: ovirt-engine >2012-09-02 13:33:57,920 INFO [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-4-thread-7) SSH execute 10.35.102.10 'cat > '/tmp/ovirt-id_rsa_f753998d-ff98-4769-849b-105c39efc630'' >2012-09-02 13:33:58,004 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-7) [5ac74306] Installation of 10.35.102.10. Executing installation stage. (Stage: Running first installation script on Host) >2012-09-02 13:33:58,004 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-7) [5ac74306] Installation of 10.35.102.10. Sending SSH Command umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar -C "${MYTMP}" --no-same-permissions -o -x && "${MYTMP}"/setup -c 'ssl=true;management_port=54321' -O 'qa.lab.tlv.redhat.com' -t 2012-09-02T10:33:58 -S /tmp/ovirt-id_rsa_f753998d-ff98-4769-849b-105c39efc630 -p 80 -b http://gadi-rhevm.qa.lab.tlv.redhat.com:80/Components/vds/ http://gadi-rhevm.qa.lab.tlv.redhat.com:80/Components/vds/ 10.35.102.10 f753998d-ff98-4769-849b-105c39efc630 False < /var/cache/ovirt-engine/vdsm-bootstrap-2.tar. (Stage: Running first installation script on Host) >2012-09-02 13:33:58,005 INFO [org.ovirt.engine.core.utils.hostinstall.CachedTar] (pool-4-thread-7) Tarball '/var/cache/ovirt-engine/vdsm-bootstrap-2.tar' is missing, creating >2012-09-02 13:33:58,027 INFO [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-4-thread-7) SSH execute 10.35.102.10 'umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar -C "${MYTMP}" --no-same-permissions -o -x && "${MYTMP}"/setup -c 'ssl=true;management_port=54321' -O 'qa.lab.tlv.redhat.com' -t 2012-09-02T10:33:58 -S /tmp/ovirt-id_rsa_f753998d-ff98-4769-849b-105c39efc630 -p 80 -b http://gadi-rhevm.qa.lab.tlv.redhat.com:80/Components/vds/ http://gadi-rhevm.qa.lab.tlv.redhat.com:80/Components/vds/ 10.35.102.10 f753998d-ff98-4769-849b-105c39efc630 False' >2012-09-02 13:33:58,899 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-3) Installation of 10.35.102.10. Received message: <BSTRAP component='INSTALLER' status='OK' message='Test platform succeeded'/>. FYI. (Stage: Running first installation script on Host) >2012-09-02 13:33:59,333 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-5) Initializing Host: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 13:33:59,992 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-3) Installation of 10.35.102.10. Received message: <BSTRAP component='SET_SYSTEM_TIME' status='OK' message='setSystemTime ended successfully'/>. FYI. (Stage: Running first installation script on Host) >2012-09-02 13:34:01,296 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-3) Installation of 10.35.102.10. Received message: <BSTRAP component='RHN_REGISTRATION' status='OK' message='Host properly registered with RHN/Satellite.'/>. FYI. (Stage: Running first installation script on Host) >2012-09-02 13:34:01,640 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-3) Installation of 10.35.102.10. Received message: <BSTRAP component='VDSM_MAJOR_VER' status='OK' message='Available VDSM matches requirements'/>. FYI. (Stage: Running first installation script on Host) >2012-09-02 13:34:01,678 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-3) Installation of 10.35.102.10. Received message: <BSTRAP component='VT_SVM' status='OK' processor='Intel' message='Server supports virtualization'/>. FYI. (Stage: Running first installation script on Host) >2012-09-02 13:34:01,711 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-3) Installation of 10.35.102.10. Received message: <BSTRAP component='OS' status='OK' type='RHEL6' message='Supported platform version'/>. FYI. (Stage: Running first installation script on Host) >2012-09-02 13:34:01,763 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-3) Installation of 10.35.102.10. Received message: <BSTRAP component='KERNEL' status='OK' version='279' message='Supported kernel version: 279'/>. FYI. (Stage: Running first installation script on Host) >2012-09-02 13:34:01,792 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-3) Installation of 10.35.102.10. Received message: <BSTRAP component='REQ PACKAGES' status='OK' result='SDL.x86_64' message='SDL-1.2.14-3.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-09-02 13:34:01,846 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-3) Installation of 10.35.102.10. Received message: <BSTRAP component='REQ PACKAGES' status='OK' result='bridge-utils.x86_64' message='bridge-utils-1.2-9.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-09-02 13:34:01,859 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-3) Installation of 10.35.102.10. Received message: <BSTRAP component='REQ PACKAGES' status='OK' result='mesa-libGLU.x86_64' message='mesa-libGLU-7.11-5.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-09-02 13:34:01,883 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-3) Installation of 10.35.102.10. Received message: <BSTRAP component='REQ PACKAGES' status='OK' result='openssl.x86_64' message='openssl-1.0.0-25.el6_3.1.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-09-02 13:34:01,955 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-3) Installation of 10.35.102.10. Received message: <BSTRAP component='REQ PACKAGES' status='OK' result='m2crypto.x86_64' message='m2crypto-0.20.2-9.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-09-02 13:34:02,057 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-3) Installation of 10.35.102.10. Received message: <BSTRAP component='REQ PACKAGES' status='OK' result='rsync.x86_64' message='rsync-3.0.6-9.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-09-02 13:34:02,076 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-3) Installation of 10.35.102.10. Received message: <BSTRAP component='VDS PACKAGES' status='WARN' result='qemu-kvm' message='package qemu-kvm is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-09-02 13:34:02,111 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-3) Installation of 10.35.102.10. Received message: <BSTRAP component='VDS PACKAGES' status='WARN' result='qemu-kvm-tools' message='package qemu-kvm-tools is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-09-02 13:34:02,115 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-3) Installation of 10.35.102.10. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='vdsm' message='vdsm-4.9.6-31.0.el6_3.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-09-02 13:34:02,136 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-3) Installation of 10.35.102.10. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='vdsm-cli' message='vdsm-cli-4.9.6-31.0.el6_3.noarch '/>. FYI. (Stage: Running first installation script on Host) >2012-09-02 13:34:02,153 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-3) Installation of 10.35.102.10. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='libjpeg' message='libjpeg-6b-46.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-09-02 13:34:02,179 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-3) Installation of 10.35.102.10. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='spice-server' message='spice-server-0.10.1-10.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-09-02 13:34:02,226 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-3) Installation of 10.35.102.10. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='pixman' message='pixman-0.18.4-1.el6_0.1.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-09-02 13:34:02,255 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-3) Installation of 10.35.102.10. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='seabios' message='seabios-0.6.1.2-19.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-09-02 13:34:02,290 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-3) Installation of 10.35.102.10. Received message: <BSTRAP component='VDS PACKAGES' status='WARN' result='qemu-img' message='package qemu-img is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-09-02 13:34:02,294 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-3) Installation of 10.35.102.10. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='fence-agents' message='fence-agents-3.1.5-17.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-09-02 13:34:02,316 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-3) Installation of 10.35.102.10. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='libselinux-python' message='libselinux-python-2.0.94-5.3.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-09-02 13:34:02,342 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-3) Installation of 10.35.102.10. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='sanlock' message='sanlock-2.3-3.el6_3.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-09-02 13:34:02,354 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-3) Installation of 10.35.102.10. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='sanlock-python' message='sanlock-python-2.3-3.el6_3.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-09-02 13:35:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-51) Autorecovering hosts is disabled, skipping >2012-09-02 13:35:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-51) Autorecovering storage domains is disabled, skipping >2012-09-02 13:35:11,635 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-3) Installation of 10.35.102.10. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='vdsm' message='vdsm-4.9.6-31.0.el6_3.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-09-02 13:35:13,990 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-3) Installation of 10.35.102.10. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='vdsm-cli' message='vdsm-cli-4.9.6-31.0.el6_3.noarch '/>. FYI. (Stage: Running first installation script on Host) >2012-09-02 13:35:16,211 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-3) Installation of 10.35.102.10. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='libjpeg' message='libjpeg-6b-46.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-09-02 13:35:18,591 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-3) Installation of 10.35.102.10. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='spice-server' message='spice-server-0.10.1-10.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-09-02 13:35:20,823 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-3) Installation of 10.35.102.10. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='pixman' message='pixman-0.18.4-1.el6_0.1.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-09-02 13:35:23,082 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-3) Installation of 10.35.102.10. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='seabios' message='seabios-0.6.1.2-19.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-09-02 13:35:25,411 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-3) Installation of 10.35.102.10. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='fence-agents' message='fence-agents-3.1.5-17.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-09-02 13:35:27,768 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-3) Installation of 10.35.102.10. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='libselinux-python' message='libselinux-python-2.0.94-5.3.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-09-02 13:35:29,991 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-3) Installation of 10.35.102.10. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='sanlock' message='sanlock-2.3-3.el6_3.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-09-02 13:35:32,338 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-3) Installation of 10.35.102.10. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='sanlock-python' message='sanlock-python-2.3-3.el6_3.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-09-02 13:35:34,705 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-3) Installation of 10.35.102.10. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='qemu-img' message='package qemu-img is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-09-02 13:35:37,049 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-3) Installation of 10.35.102.10. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='qemu-kvm-tools' message='package qemu-kvm-tools is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-09-02 13:35:39,373 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-3) Installation of 10.35.102.10. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='qemu-kvm' message='package qemu-kvm is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-09-02 13:35:39,394 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-3) Installation of 10.35.102.10. Received message: <BSTRAP component='CreateConf' status='OK' message='Basic configuration found, skipping this step'/>. FYI. (Stage: Running first installation script on Host) >2012-09-02 13:35:39,405 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-3) Installation of 10.35.102.10. Received message: <BSTRAP component='CheckLocalHostname' status='OK' message='Local hostname is correct.'/>. FYI. (Stage: Running first installation script on Host) >2012-09-02 13:35:39,458 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-3) Installation of 10.35.102.10. Received message: <BSTRAP component='SetNetworking' status='OK' message='Bridge management already exists. Skipping bridge creation.'/>. FYI. (Stage: Running first installation script on Host) >2012-09-02 13:35:39,578 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-3) Installation of 10.35.102.10. Received message: <BSTRAP component='SetSSHAccess' status='OK' message='SUCCESS'/>. FYI. (Stage: Running first installation script on Host) >2012-09-02 13:35:40,010 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-3) Installation of 10.35.102.10. Received message: <BSTRAP component='VerifyServices' status='OK' message='Needed services set'/>. FYI. (Stage: Running first installation script on Host) >2012-09-02 13:35:44,786 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-3) Installation of 10.35.102.10. Received message: <BSTRAP component='Encryption setup' status='OK' message='Ended successfully'/>. FYI. (Stage: Running first installation script on Host) >2012-09-02 13:35:44,985 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-3) Installation of 10.35.102.10. Received message: <BSTRAP component='RHEV_INSTALL' status='OK'/>. Stage completed. (Stage: Running first installation script on Host) >2012-09-02 13:35:45,132 ERROR [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-7) [5ac74306] Installation of 10.35.102.10. Error: Repository RHEVM-Build is listed more than once in the configuration >Repository KIPI is listed more than once in the configuration >Repository JBOSS is listed more than once in the configuration >Repository Missing-Package is listed more than once in the configuration >Repository TLV is listed more than once in the configuration >Repository qa is listed more than once in the configuration >Repository rhel-source is listed more than once in the configuration >Repository rhel-source-beta is listed more than once in the configuration >Repository RHEVM-Build is listed more than once in the configuration >Repository KIPI is listed more than once in the configuration >Repository JBOSS is listed more than once in the configuration >Repository Missing-Package is listed more than once in the configuration >Repository TLV is listed more than once in the configuration >Repository qa is listed more than once in the configuration >Repository rhel-source is listed more than once in the configuration >Repository rhel-source-beta is liste. (Stage: Downloading certificate request from Host) >2012-09-02 13:35:45,133 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-7) [5ac74306] VDS message: Repository RHEVM-Build is listed more than once in the configuration >2012-09-02 13:35:45,133 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-7) [5ac74306] VDS message: Repository KIPI is listed more than once in the configuration >2012-09-02 13:35:45,133 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-7) [5ac74306] VDS message: Repository JBOSS is listed more than once in the configuration >2012-09-02 13:35:45,133 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-7) [5ac74306] VDS message: Repository Missing-Package is listed more than once in the configuration >2012-09-02 13:35:45,133 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-7) [5ac74306] VDS message: Repository TLV is listed more than once in the configuration >2012-09-02 13:35:45,133 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-7) [5ac74306] VDS message: Repository qa is listed more than once in the configuration >2012-09-02 13:35:45,133 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-7) [5ac74306] VDS message: Repository rhel-source is listed more than once in the configuration >2012-09-02 13:35:45,133 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-7) [5ac74306] VDS message: Repository rhel-source-beta is listed more than once in the configuration >2012-09-02 13:35:45,133 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-7) [5ac74306] VDS message: Repository RHEVM-Build is listed more than once in the configuration >2012-09-02 13:35:45,133 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-7) [5ac74306] VDS message: Repository KIPI is listed more than once in the configuration >2012-09-02 13:35:45,133 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-7) [5ac74306] VDS message: Repository JBOSS is listed more than once in the configuration >2012-09-02 13:35:45,133 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-7) [5ac74306] VDS message: Repository Missing-Package is listed more than once in the configuration >2012-09-02 13:35:45,133 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-7) [5ac74306] VDS message: Repository TLV is listed more than once in the configuration >2012-09-02 13:35:45,133 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-7) [5ac74306] VDS message: Repository qa is listed more than once in the configuration >2012-09-02 13:35:45,134 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-7) [5ac74306] VDS message: Repository rhel-source is listed more than once in the configuration >2012-09-02 13:35:45,134 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-7) [5ac74306] VDS message: Repository rhel-source-beta is liste >2012-09-02 13:35:45,134 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-7) [5ac74306] Script ended, result is {1} >2012-09-02 13:35:45,134 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-7) [5ac74306] Installation of 10.35.102.10. Executing installation stage. (Stage: Downloading certificate request from Host) >2012-09-02 13:35:45,134 INFO [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-4-thread-7) SSH receive 10.35.102.10:'/tmp/cert_f753998d-ff98-4769-849b-105c39efc630.req' '/etc/pki/ovirt-engine/requests/cert_f753998d-ff98-4769-849b-105c39efc630.req') >2012-09-02 13:35:45,168 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-7) [5ac74306] Installation of 10.35.102.10. successfully done sftp operation ( Stage: Downloading certificate request from Host) >2012-09-02 13:35:45,173 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-7) [5ac74306] DownloadCertificateRequest ended:true >2012-09-02 13:35:45,173 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-7) [5ac74306] Installation of 10.35.102.10. Executing installation stage. (Stage: Sign certificate request and generate certificate) >2012-09-02 13:35:46,287 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-7) [5ac74306] SignCertificateRequest ended:true >2012-09-02 13:35:46,287 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-7) [5ac74306] Installation of 10.35.102.10. Executing installation stage. (Stage: Upload signed sertificate to Host) >2012-09-02 13:35:46,309 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-7) [5ac74306] Installation of 10.35.102.10. successfully done sftp operation ( Stage: Upload signed sertificate to Host) >2012-09-02 13:35:46,309 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-7) [5ac74306] UploadSignedCertificate ended:true >2012-09-02 13:35:46,309 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-7) [5ac74306] Installation of 10.35.102.10. Executing installation stage. (Stage: Upload Cerficate Autority to Host) >2012-09-02 13:35:46,364 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-7) [5ac74306] Installation of 10.35.102.10. successfully done sftp operation ( Stage: Upload Cerficate Autority to Host) >2012-09-02 13:35:46,364 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-7) [5ac74306] Installation of 10.35.102.10. Executing installation stage. (Stage: Running second installation script on Host) >2012-09-02 13:35:46,364 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-7) [5ac74306] Installation of 10.35.102.10. Sending SSH Command umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar -C "${MYTMP}" --no-same-permissions -o -x && "${MYTMP}"/setup -c 'ssl=true;management_port=54321' -O 'qa.lab.tlv.redhat.com' -t 2012-09-02T10:33:58 -S /tmp/ovirt-id_rsa_f753998d-ff98-4769-849b-105c39efc630 -p 80 -b http://gadi-rhevm.qa.lab.tlv.redhat.com:80/Components/vds/ http://gadi-rhevm.qa.lab.tlv.redhat.com:80/Components/vds/ 10.35.102.10 f753998d-ff98-4769-849b-105c39efc630 True < /var/cache/ovirt-engine/vdsm-bootstrap-2.tar. (Stage: Running second installation script on Host) >2012-09-02 13:35:46,364 INFO [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-4-thread-7) SSH execute 10.35.102.10 'umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar -C "${MYTMP}" --no-same-permissions -o -x && "${MYTMP}"/setup -c 'ssl=true;management_port=54321' -O 'qa.lab.tlv.redhat.com' -t 2012-09-02T10:33:58 -S /tmp/ovirt-id_rsa_f753998d-ff98-4769-849b-105c39efc630 -p 80 -b http://gadi-rhevm.qa.lab.tlv.redhat.com:80/Components/vds/ http://gadi-rhevm.qa.lab.tlv.redhat.com:80/Components/vds/ 10.35.102.10 f753998d-ff98-4769-849b-105c39efc630 True' >2012-09-02 13:35:46,661 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-3) Installation of 10.35.102.10. Received message: <BSTRAP component='INSTALLER' status='OK' message='Test platform succeeded'/>. FYI. (Stage: Running second installation script on Host) >2012-09-02 13:36:16,337 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-3) Installation of 10.35.102.10. Received message: <BSTRAP component='instCert' status='OK'/>. FYI. (Stage: Running second installation script on Host) >2012-09-02 13:36:16,781 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-3) Installation of 10.35.102.10. Received message: <BSTRAP component='CoreDump' status='OK'/>. FYI. (Stage: Running second installation script on Host) >2012-09-02 13:36:16,803 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-3) Installation of 10.35.102.10. Received message: <BSTRAP component='cleanAll' status='OK'/>. FYI. (Stage: Running second installation script on Host) >2012-09-02 13:36:16,821 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-3) Installation of 10.35.102.10. Received message: <BSTRAP component='VDS Configuration' status='OK'/>. FYI. (Stage: Running second installation script on Host) >2012-09-02 13:36:16,840 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-3) Installation of 10.35.102.10. Received message: <BSTRAP component='RHEV_INSTALL' status='OK'/>. Stage completed. (Stage: Running second installation script on Host) >2012-09-02 13:36:16,858 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-3) Installation of 10.35.102.10. Received message: <BSTRAP component='Reboot' status='OK' message='Rebooting machine' />. FYI. (Stage: Host installation complete) >2012-09-02 13:36:16,870 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-7) [5ac74306] Script ended, result is {1} >2012-09-02 13:36:16,870 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-4-thread-7) [5ac74306] After Installation pool-4-thread-7 >2012-09-02 13:36:16,872 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-7) [5ac74306] START, SetVdsStatusVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, status=Reboot, nonOperationalReason=NONE), log id: 315897c8 >2012-09-02 13:36:16,926 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-7) [5ac74306] FINISH, SetVdsStatusVDSCommand, log id: 315897c8 >2012-09-02 13:36:16,932 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-4-thread-8) [5ac74306] Waiting 300 seconds, for server to finish reboot process. >2012-09-02 13:40:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-78) Autorecovering hosts is disabled, skipping >2012-09-02 13:40:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-78) Autorecovering storage domains is disabled, skipping >2012-09-02 13:41:16,935 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-8) [5ac74306] START, SetVdsStatusVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, status=NonResponsive, nonOperationalReason=NONE), log id: 75310531 >2012-09-02 13:41:17,006 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-8) [5ac74306] FINISH, SetVdsStatusVDSCommand, log id: 75310531 >2012-09-02 13:41:21,213 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-37) [9ca026b] Running command: InitVdsOnUpCommand internal: true. >2012-09-02 13:41:21,242 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-37) [9ca026b] No string for UNASSIGNED type. Use default Log >2012-09-02 13:41:21,273 INFO [org.ovirt.engine.core.bll.MultipleActionsRunner] (pool-4-thread-12) [9ca026b] MultipleActionsRunner of type MigrateVmToServer invoked with no actions >2012-09-02 13:41:21,306 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-37) [5e1df278] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: b32eacf0-f4e9-11e1-973c-001a4a169705 Type: VDS >2012-09-02 13:41:21,337 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-37) [9670cc5] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: b32eacf0-f4e9-11e1-973c-001a4a169705 Type: VDS >2012-09-02 13:41:21,338 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-37) [9670cc5] START, SetVdsStatusVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, status=NonOperational, nonOperationalReason=CPU_TYPE_INCOMPATIBLE_WITH_CLUSTER), log id: 7d640a87 >2012-09-02 13:41:21,342 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-37) [9670cc5] FINISH, SetVdsStatusVDSCommand, log id: 7d640a87 >2012-09-02 13:41:21,502 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-37) [36253582] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: b32eacf0-f4e9-11e1-973c-001a4a169705 Type: VDS >2012-09-02 13:41:21,506 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-37) [36253582] Host b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2012-09-02 13:44:40,792 INFO [org.ovirt.engine.core.bll.storage.FenceVdsManualyCommand] (ajp-/127.0.0.1:8009-1) Lock Acquired to object EngineLock [exclusiveLocks= key: b32eacf0-f4e9-11e1-973c-001a4a169705 value: VDS_FENCE >, sharedLocks= ] >2012-09-02 13:44:40,795 INFO [org.ovirt.engine.core.bll.storage.FenceVdsManualyCommand] (pool-4-thread-14) [2c1fcabb] Running command: FenceVdsManualyCommand internal: false. Entities affected : ID: b32eacf0-f4e9-11e1-973c-001a4a169705 Type: VDS >2012-09-02 13:44:40,843 INFO [org.ovirt.engine.core.bll.ClearNonResponsiveVdsVmsCommand] (pool-4-thread-14) [31f2ab74] Running command: ClearNonResponsiveVdsVmsCommand internal: true. Entities affected : ID: b32eacf0-f4e9-11e1-973c-001a4a169705 Type: VDS >2012-09-02 13:44:40,863 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsVMsClearedVDSCommand] (pool-4-thread-14) [31f2ab74] START, UpdateVdsVMsClearedVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705), log id: 13a07710 >2012-09-02 13:44:40,868 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsVMsClearedVDSCommand] (pool-4-thread-14) [31f2ab74] FINISH, UpdateVdsVMsClearedVDSCommand, log id: 13a07710 >2012-09-02 13:44:40,908 INFO [org.ovirt.engine.core.bll.storage.FenceVdsManualyCommand] (pool-4-thread-14) [31f2ab74] Lock freed to object EngineLock [exclusiveLocks= key: b32eacf0-f4e9-11e1-973c-001a4a169705 value: VDS_FENCE >, sharedLocks= ] >2012-09-02 13:45:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-4) Autorecovering hosts is disabled, skipping >2012-09-02 13:45:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-4) Autorecovering storage domains is disabled, skipping >2012-09-02 13:46:12,891 INFO [org.ovirt.engine.core.bll.storage.FenceVdsManualyCommand] (ajp-/127.0.0.1:8009-6) Lock Acquired to object EngineLock [exclusiveLocks= key: b32eacf0-f4e9-11e1-973c-001a4a169705 value: VDS_FENCE >, sharedLocks= ] >2012-09-02 13:46:12,897 INFO [org.ovirt.engine.core.bll.storage.FenceVdsManualyCommand] (pool-4-thread-15) [5d833671] Running command: FenceVdsManualyCommand internal: false. Entities affected : ID: b32eacf0-f4e9-11e1-973c-001a4a169705 Type: VDS >2012-09-02 13:46:12,902 INFO [org.ovirt.engine.core.bll.ClearNonResponsiveVdsVmsCommand] (pool-4-thread-15) [76850ab4] Running command: ClearNonResponsiveVdsVmsCommand internal: true. Entities affected : ID: b32eacf0-f4e9-11e1-973c-001a4a169705 Type: VDS >2012-09-02 13:46:12,951 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsVMsClearedVDSCommand] (pool-4-thread-15) [76850ab4] START, UpdateVdsVMsClearedVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705), log id: 3ad9f68e >2012-09-02 13:46:12,954 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsVMsClearedVDSCommand] (pool-4-thread-15) [76850ab4] FINISH, UpdateVdsVMsClearedVDSCommand, log id: 3ad9f68e >2012-09-02 13:46:12,978 INFO [org.ovirt.engine.core.bll.storage.FenceVdsManualyCommand] (pool-4-thread-15) [76850ab4] Lock freed to object EngineLock [exclusiveLocks= key: b32eacf0-f4e9-11e1-973c-001a4a169705 value: VDS_FENCE >, sharedLocks= ] >2012-09-02 13:49:58,684 INFO [org.ovirt.engine.core.bll.MaintananceNumberOfVdssCommand] (pool-4-thread-16) [3b558d3a] Running command: MaintananceNumberOfVdssCommand internal: false. Entities affected : ID: b32eacf0-f4e9-11e1-973c-001a4a169705 Type: VDS >2012-09-02 13:49:58,687 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-16) [3b558d3a] START, SetVdsStatusVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, status=PreparingForMaintenance, nonOperationalReason=NONE), log id: 1bc25810 >2012-09-02 13:49:58,723 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-16) [3b558d3a] FINISH, SetVdsStatusVDSCommand, log id: 1bc25810 >2012-09-02 13:49:58,806 INFO [org.ovirt.engine.core.bll.MaintananceVdsCommand] (pool-4-thread-16) [3b558d3a] Running command: MaintananceVdsCommand internal: true. Entities affected : ID: b32eacf0-f4e9-11e1-973c-001a4a169705 Type: VDS >2012-09-02 13:49:58,859 WARN [org.ovirt.engine.core.dal.dbbroker.generic.DBConfigUtils] (pool-4-thread-16) [3b558d3a] Adding new value to configuration cache. >2012-09-02 13:49:58,859 WARN [org.ovirt.engine.core.dal.dbbroker.generic.DBConfigUtils] (pool-4-thread-16) [3b558d3a] Didnt find the value of DBI18NPrefix in DB for version general - using default: >2012-09-02 13:49:59,059 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-29) vds::Updated vds status from Preparing for Maintenance to Maintenance in database, vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 13:50:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-30) Autorecovering hosts is disabled, skipping >2012-09-02 13:50:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-30) Autorecovering storage domains is disabled, skipping >2012-09-02 13:50:00,247 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (ajp-/127.0.0.1:8009-1) Lock Acquired to object EngineLock [exclusiveLocks= key: b32eacf0-f4e9-11e1-973c-001a4a169705 value: VDS >, sharedLocks= ] >2012-09-02 13:50:00,322 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (pool-4-thread-19) [2c8119b9] Running command: ActivateVdsCommand internal: false. Entities affected : ID: b32eacf0-f4e9-11e1-973c-001a4a169705 Type: VDS >2012-09-02 13:50:00,360 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-19) [2c8119b9] START, SetVdsStatusVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, status=Unassigned, nonOperationalReason=NONE), log id: 6d2b9371 >2012-09-02 13:50:00,372 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-19) [2c8119b9] FINISH, SetVdsStatusVDSCommand, log id: 6d2b9371 >2012-09-02 13:50:00,386 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (pool-4-thread-19) [2c8119b9] START, ActivateVdsVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705), log id: 73821850 >2012-09-02 13:50:00,838 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (pool-4-thread-19) [71c60ef8] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: b32eacf0-f4e9-11e1-973c-001a4a169705 Type: VDS >2012-09-02 13:50:00,842 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (pool-4-thread-19) [5a1e3957] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: b32eacf0-f4e9-11e1-973c-001a4a169705 Type: VDS >2012-09-02 13:50:00,842 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-19) [5a1e3957] START, SetVdsStatusVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, status=NonOperational, nonOperationalReason=CPU_TYPE_INCOMPATIBLE_WITH_CLUSTER), log id: 73c7ff3 >2012-09-02 13:50:00,855 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-19) [5a1e3957] FINISH, SetVdsStatusVDSCommand, log id: 73c7ff3 >2012-09-02 13:50:00,905 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (pool-4-thread-19) [3bdcf2a7] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: b32eacf0-f4e9-11e1-973c-001a4a169705 Type: VDS >2012-09-02 13:50:00,907 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (pool-4-thread-19) [3bdcf2a7] FINISH, ActivateVdsVDSCommand, log id: 73821850 >2012-09-02 13:50:00,945 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (pool-4-thread-19) [3bdcf2a7] Lock freed to object EngineLock [exclusiveLocks= key: b32eacf0-f4e9-11e1-973c-001a4a169705 value: VDS >, sharedLocks= ] >2012-09-02 13:53:47,543 WARN [org.ovirt.engine.core.bll.UpdateVdsGroupCommand] (ajp-/127.0.0.1:8009-7) [1e1b4232] CanDoAction of action UpdateVdsGroup failed. Reasons:VAR__TYPE__CLUSTER,VAR__ACTION__UPDATE,VDS_GROUP_CANNOT_UPDATE_CPU_ILLEGAL >2012-09-02 13:53:59,773 INFO [org.ovirt.engine.core.bll.MaintananceNumberOfVdssCommand] (pool-4-thread-22) [65dda839] Running command: MaintananceNumberOfVdssCommand internal: false. Entities affected : ID: b32eacf0-f4e9-11e1-973c-001a4a169705 Type: VDS >2012-09-02 13:53:59,774 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-22) [65dda839] START, SetVdsStatusVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, status=PreparingForMaintenance, nonOperationalReason=NONE), log id: 54ec4bb2 >2012-09-02 13:53:59,927 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-22) [65dda839] FINISH, SetVdsStatusVDSCommand, log id: 54ec4bb2 >2012-09-02 13:54:00,001 INFO [org.ovirt.engine.core.bll.MaintananceVdsCommand] (pool-4-thread-22) [65dda839] Running command: MaintananceVdsCommand internal: true. Entities affected : ID: b32eacf0-f4e9-11e1-973c-001a4a169705 Type: VDS >2012-09-02 13:54:00,750 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-12) vds::Updated vds status from Preparing for Maintenance to Maintenance in database, vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 13:54:09,099 INFO [org.ovirt.engine.core.bll.ChangeVDSClusterCommand] (ajp-/127.0.0.1:8009-2) [33de50a7] Running command: ChangeVDSClusterCommand internal: false. Entities affected : ID: b32eacf0-f4e9-11e1-973c-001a4a169705 Type: VDS, ID: 99408929-82cf-4dc7-a532-9d998063fa95 Type: VdsGroups >2012-09-02 13:54:09,173 INFO [org.ovirt.engine.core.bll.RemoveVdsSpmIdCommand] (ajp-/127.0.0.1:8009-2) [33de50a7] Running command: RemoveVdsSpmIdCommand internal: true. Entities affected : ID: b32eacf0-f4e9-11e1-973c-001a4a169705 Type: VDS >2012-09-02 13:54:09,197 INFO [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp-/127.0.0.1:8009-2) [33de50a7] Running command: AddVdsSpmIdCommand internal: true. Entities affected : ID: b32eacf0-f4e9-11e1-973c-001a4a169705 Type: VDS >2012-09-02 13:54:09,224 INFO [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp-/127.0.0.1:8009-2) [33de50a7] Lock freed to object EngineLock [exclusiveLocks= key: c7387742-f4dc-11e1-8ca7-001a4a169705 value: REGISTER_VDS >, sharedLocks= ] >2012-09-02 13:54:09,280 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp-/127.0.0.1:8009-3) [76fc066e] Running command: UpdateVdsCommand internal: false. Entities affected : ID: b32eacf0-f4e9-11e1-973c-001a4a169705 Type: VDS >2012-09-02 13:54:15,103 INFO [org.ovirt.engine.core.bll.UpdateVdsGroupCommand] (ajp-/127.0.0.1:8009-2) [105e4e3c] Running command: UpdateVdsGroupCommand internal: false. Entities affected : ID: b1600626-f4e9-11e1-b9cc-001a4a169705 Type: VdsGroups >2012-09-02 13:54:25,092 INFO [org.ovirt.engine.core.bll.ChangeVDSClusterCommand] (ajp-/127.0.0.1:8009-7) [15ab5d60] Running command: ChangeVDSClusterCommand internal: false. Entities affected : ID: b32eacf0-f4e9-11e1-973c-001a4a169705 Type: VDS, ID: b1600626-f4e9-11e1-b9cc-001a4a169705 Type: VdsGroups >2012-09-02 13:54:25,131 INFO [org.ovirt.engine.core.bll.RemoveVdsSpmIdCommand] (ajp-/127.0.0.1:8009-7) [15ab5d60] Running command: RemoveVdsSpmIdCommand internal: true. Entities affected : ID: b32eacf0-f4e9-11e1-973c-001a4a169705 Type: VDS >2012-09-02 13:54:25,160 INFO [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp-/127.0.0.1:8009-7) [15ab5d60] Running command: AddVdsSpmIdCommand internal: true. Entities affected : ID: b32eacf0-f4e9-11e1-973c-001a4a169705 Type: VDS >2012-09-02 13:54:25,182 INFO [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp-/127.0.0.1:8009-7) [15ab5d60] Lock freed to object EngineLock [exclusiveLocks= key: ba2bec5d-7b0c-4fbc-941f-a2a62457907a value: REGISTER_VDS >, sharedLocks= ] >2012-09-02 13:54:25,234 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp-/127.0.0.1:8009-1) [6303a071] Running command: UpdateVdsCommand internal: false. Entities affected : ID: b32eacf0-f4e9-11e1-973c-001a4a169705 Type: VDS >2012-09-02 13:54:29,654 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (ajp-/127.0.0.1:8009-3) Lock Acquired to object EngineLock [exclusiveLocks= key: b32eacf0-f4e9-11e1-973c-001a4a169705 value: VDS >, sharedLocks= ] >2012-09-02 13:54:29,947 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (pool-4-thread-25) [2cdecf11] Running command: ActivateVdsCommand internal: false. Entities affected : ID: b32eacf0-f4e9-11e1-973c-001a4a169705 Type: VDS >2012-09-02 13:54:29,980 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-25) [2cdecf11] START, SetVdsStatusVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, status=Unassigned, nonOperationalReason=NONE), log id: 6de4b0 >2012-09-02 13:54:29,987 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-25) [2cdecf11] FINISH, SetVdsStatusVDSCommand, log id: 6de4b0 >2012-09-02 13:54:30,008 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (pool-4-thread-25) [2cdecf11] START, ActivateVdsVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705), log id: 6a78c189 >2012-09-02 13:54:30,848 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (pool-4-thread-25) [5f21a475] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: b32eacf0-f4e9-11e1-973c-001a4a169705 Type: VDS >2012-09-02 13:54:30,850 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (pool-4-thread-25) [3a8f7863] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: b32eacf0-f4e9-11e1-973c-001a4a169705 Type: VDS >2012-09-02 13:54:30,852 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (pool-4-thread-25) [3a8f7863] FINISH, ActivateVdsVDSCommand, log id: 6a78c189 >2012-09-02 13:54:30,885 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (pool-4-thread-25) [3a8f7863] Lock freed to object EngineLock [exclusiveLocks= key: b32eacf0-f4e9-11e1-973c-001a4a169705 value: VDS >, sharedLocks= ] >2012-09-02 13:54:31,269 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-35) [7bdd084] Running command: InitVdsOnUpCommand internal: true. >2012-09-02 13:54:31,287 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-35) [7bdd084] No string for UNASSIGNED type. Use default Log >2012-09-02 13:54:31,301 INFO [org.ovirt.engine.core.bll.MultipleActionsRunner] (pool-4-thread-29) [7bdd084] MultipleActionsRunner of type MigrateVmToServer invoked with no actions >2012-09-02 13:54:31,305 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-35) [7e632e98] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: b32eacf0-f4e9-11e1-973c-001a4a169705 Type: VDS >2012-09-02 13:54:31,336 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-35) [372f1ff9] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: b32eacf0-f4e9-11e1-973c-001a4a169705 Type: VDS >2012-09-02 13:55:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-55) Autorecovering hosts is disabled, skipping >2012-09-02 13:55:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-55) Autorecovering storage domains is disabled, skipping >2012-09-02 14:00:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-78) Autorecovering hosts is disabled, skipping >2012-09-02 14:00:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-78) Autorecovering storage domains is disabled, skipping >2012-09-02 14:02:18,354 INFO [org.ovirt.engine.core.bll.MaintananceNumberOfVdssCommand] (pool-4-thread-49) [6b8a59c2] Running command: MaintananceNumberOfVdssCommand internal: false. Entities affected : ID: b32eacf0-f4e9-11e1-973c-001a4a169705 Type: VDS >2012-09-02 14:02:18,355 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-49) [6b8a59c2] START, SetVdsStatusVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, status=PreparingForMaintenance, nonOperationalReason=NONE), log id: 1e3f4fe9 >2012-09-02 14:02:18,385 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-49) [6b8a59c2] FINISH, SetVdsStatusVDSCommand, log id: 1e3f4fe9 >2012-09-02 14:02:18,451 INFO [org.ovirt.engine.core.bll.MaintananceVdsCommand] (pool-4-thread-49) [6b8a59c2] Running command: MaintananceVdsCommand internal: true. Entities affected : ID: b32eacf0-f4e9-11e1-973c-001a4a169705 Type: VDS >2012-09-02 14:02:19,409 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-82) vds::Updated vds status from Preparing for Maintenance to Maintenance in database, vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 14:02:19,829 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (ajp-/127.0.0.1:8009-4) Lock Acquired to object EngineLock [exclusiveLocks= key: b32eacf0-f4e9-11e1-973c-001a4a169705 value: VDS >, sharedLocks= ] >2012-09-02 14:02:19,949 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (pool-4-thread-50) [61ad9d31] Running command: ActivateVdsCommand internal: false. Entities affected : ID: b32eacf0-f4e9-11e1-973c-001a4a169705 Type: VDS >2012-09-02 14:02:20,002 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-50) [61ad9d31] START, SetVdsStatusVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, status=Unassigned, nonOperationalReason=NONE), log id: ea3e392 >2012-09-02 14:02:20,005 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-50) [61ad9d31] FINISH, SetVdsStatusVDSCommand, log id: ea3e392 >2012-09-02 14:02:20,015 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (pool-4-thread-50) [61ad9d31] START, ActivateVdsVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705), log id: 33d5157f >2012-09-02 14:02:20,303 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (pool-4-thread-50) [f3f59be] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: b32eacf0-f4e9-11e1-973c-001a4a169705 Type: VDS >2012-09-02 14:02:20,307 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (pool-4-thread-50) [45130a3b] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: b32eacf0-f4e9-11e1-973c-001a4a169705 Type: VDS >2012-09-02 14:02:20,308 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (pool-4-thread-50) [45130a3b] FINISH, ActivateVdsVDSCommand, log id: 33d5157f >2012-09-02 14:02:20,332 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (pool-4-thread-50) [45130a3b] Lock freed to object EngineLock [exclusiveLocks= key: b32eacf0-f4e9-11e1-973c-001a4a169705 value: VDS >, sharedLocks= ] >2012-09-02 14:02:21,777 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-83) [f6c33a6] Running command: InitVdsOnUpCommand internal: true. >2012-09-02 14:02:21,784 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-83) [f6c33a6] No string for UNASSIGNED type. Use default Log >2012-09-02 14:02:21,785 INFO [org.ovirt.engine.core.bll.MultipleActionsRunner] (pool-4-thread-50) [f6c33a6] MultipleActionsRunner of type MigrateVmToServer invoked with no actions >2012-09-02 14:02:21,811 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-83) [5f81ae08] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: b32eacf0-f4e9-11e1-973c-001a4a169705 Type: VDS >2012-09-02 14:02:21,830 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-83) [28a39dff] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: b32eacf0-f4e9-11e1-973c-001a4a169705 Type: VDS >2012-09-02 14:03:41,963 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-44) DbUserCacheManager::refreshAllUserData() - entered >2012-09-02 14:05:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-4) Autorecovering hosts is disabled, skipping >2012-09-02 14:05:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-4) Autorecovering storage domains is disabled, skipping >2012-09-02 14:10:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-28) Autorecovering hosts is disabled, skipping >2012-09-02 14:10:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-28) Autorecovering storage domains is disabled, skipping >2012-09-02 14:15:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-51) Autorecovering hosts is disabled, skipping >2012-09-02 14:15:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-51) Autorecovering storage domains is disabled, skipping >2012-09-02 14:20:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-75) Autorecovering hosts is disabled, skipping >2012-09-02 14:20:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-75) Autorecovering storage domains is disabled, skipping >2012-09-02 14:20:36,498 INFO [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (ajp-/127.0.0.1:8009-6) SSH execute 10.35.116.2 'dmidecode | awk ' /UUID/{ print $2; } ' | tr ' >' '_' && cat /sys/class/net/*/address | sed -e '/00:00:00:00/d' -e '/^$/d' | sort -u | head -n 1' >2012-09-02 14:20:37,504 INFO [org.ovirt.engine.core.bll.AddVdsCommand] (ajp-/127.0.0.1:8009-6) [7782aea3] Running command: AddVdsCommand internal: false. Entities affected : ID: 99408929-82cf-4dc7-a532-9d998063fa95 Type: VdsGroups >2012-09-02 14:20:37,569 INFO [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp-/127.0.0.1:8009-6) [245919eb] Running command: AddVdsSpmIdCommand internal: true. Entities affected : ID: 3880c572-f4f0-11e1-9a52-001a4a169705 Type: VDS >2012-09-02 14:20:37,646 INFO [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp-/127.0.0.1:8009-6) [245919eb] Lock freed to object EngineLock [exclusiveLocks= key: c7387742-f4dc-11e1-8ca7-001a4a169705 value: REGISTER_VDS >, sharedLocks= ] >2012-09-02 14:20:37,647 ERROR [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp-/127.0.0.1:8009-6) [245919eb] Cannot get vdsManager for vdsid=3880c572-f4f0-11e1-9a52-001a4a169705 >2012-09-02 14:20:37,647 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (ajp-/127.0.0.1:8009-6) [245919eb] START, RemoveVdsVDSCommand(vdsId = 3880c572-f4f0-11e1-9a52-001a4a169705), log id: 60117ab0 >2012-09-02 14:20:37,647 ERROR [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp-/127.0.0.1:8009-6) [245919eb] Cannot get vdsManager for vdsid=3880c572-f4f0-11e1-9a52-001a4a169705 >2012-09-02 14:20:37,647 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (ajp-/127.0.0.1:8009-6) [245919eb] FINISH, RemoveVdsVDSCommand, log id: 60117ab0 >2012-09-02 14:20:37,647 ERROR [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp-/127.0.0.1:8009-6) [245919eb] Cannot get vdsManager for vdsid=3880c572-f4f0-11e1-9a52-001a4a169705 >2012-09-02 14:20:37,647 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8009-6) [245919eb] START, AddVdsVDSCommand(vdsId = 3880c572-f4f0-11e1-9a52-001a4a169705), log id: 1b8dea16 >2012-09-02 14:20:37,647 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8009-6) [245919eb] AddVds - entered , starting logic to add VDS 3880c572-f4f0-11e1-9a52-001a4a169705 >2012-09-02 14:20:37,653 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8009-6) [245919eb] AddVds - VDS 3880c572-f4f0-11e1-9a52-001a4a169705 was added, will try to add it to the resource manager >2012-09-02 14:20:37,656 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ajp-/127.0.0.1:8009-6) [245919eb] Eneterd VdsManager:constructor >2012-09-02 14:20:37,656 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ajp-/127.0.0.1:8009-6) [245919eb] vdsBroker(10.35.116.2,54,321) >2012-09-02 14:20:37,659 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp-/127.0.0.1:8009-6) [245919eb] ResourceManager::AddVds - VDS 3880c572-f4f0-11e1-9a52-001a4a169705 was added to the Resource Manager >2012-09-02 14:20:37,659 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8009-6) [245919eb] FINISH, AddVdsVDSCommand, log id: 1b8dea16 >2012-09-02 14:20:37,794 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-4-thread-50) [6daa7da2] Running command: InstallVdsCommand internal: true. Entities affected : ID: 3880c572-f4f0-11e1-9a52-001a4a169705 Type: VDS >2012-09-02 14:20:37,795 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-4-thread-50) [6daa7da2] Before Installation pool-4-thread-50 >2012-09-02 14:20:37,795 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-50) [6daa7da2] Installation of 10.35.116.2. Executing installation stage. (Stage: Starting Host installation) >2012-09-02 14:20:37,796 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-50) [6daa7da2] Installation of 10.35.116.2. Executing installation stage. (Stage: Connecting to Host) >2012-09-02 14:20:37,937 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-50) [6daa7da2] Installation of 10.35.116.2. Received message: <BSTRAP component='RHEV_INSTALL' status='OK' message='Connected to Host 10.35.116.2 with SSH key fingerprint: 28:c2:5e:e8:fe:c9:be:26:1f:03:44:4e:5b:e0:2e:42'/>. FYI. (Stage: Connecting to Host) >2012-09-02 14:20:37,981 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-50) [6daa7da2] Installation of 10.35.116.2. Successfully connected to server ssh. (Stage: Connecting to Host) >2012-09-02 14:20:37,981 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-50) [6daa7da2] Installation of 10.35.116.2. Executing installation stage. (Stage: Get the unique vds id) >2012-09-02 14:20:37,981 INFO [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-4-thread-50) SSH execute 10.35.116.2 'dmidecode | awk ' /UUID/{ print $2; } ' | tr ' >' '_' && cat /sys/class/net/*/address | sed -e '/00:00:00:00/d' -e '/^$/d' | sort -u | head -n 1' >2012-09-02 14:20:38,165 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-7) Installation of 10.35.116.2. Received message: 6A7E2852-C2CC-11DE-9297-00215E993FDC_Number:_00:21:5e:99:3f:dc. FYI. (Stage: Get the unique vds id) >2012-09-02 14:20:38,174 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-7) Installation of 10.35.116.2. Assigning unique id 6A7E2852-C2CC-11DE-9297-00215E993FDC_Number:_00:21:5e:99:3f:dc to Host. (Stage: Get the unique vds id) >2012-09-02 14:20:38,208 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-50) [6daa7da2] Installation of 10.35.116.2. Executing installation stage. (Stage: Upload Installation script to Host) >2012-09-02 14:20:38,209 INFO [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-4-thread-50) SSH execute 10.35.116.2 'cat > '/tmp/ovirt-id_rsa_8c13abcf-d106-4b6c-99a6-f2d0c0162f64'' >2012-09-02 14:20:38,298 INFO [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-4-thread-50) SSH execute 10.35.116.2 'cat > '/tmp/firewall.conf.8c13abcf-d106-4b6c-99a6-f2d0c0162f64'' >2012-09-02 14:20:38,383 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-50) [6daa7da2] Installation of 10.35.116.2. Executing installation stage. (Stage: Running first installation script on Host) >2012-09-02 14:20:38,384 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-50) [6daa7da2] Installation of 10.35.116.2. Sending SSH Command umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar -C "${MYTMP}" --no-same-permissions -o -x && "${MYTMP}"/setup -c 'ssl=true;management_port=54321' -O 'qa.lab.tlv.redhat.com' -t 2012-09-02T11:20:38 -f /tmp/firewall.conf.8c13abcf-d106-4b6c-99a6-f2d0c0162f64 -S /tmp/ovirt-id_rsa_8c13abcf-d106-4b6c-99a6-f2d0c0162f64 -p 80 -b http://gadi-rhevm.qa.lab.tlv.redhat.com:80/Components/vds/ http://gadi-rhevm.qa.lab.tlv.redhat.com:80/Components/vds/ 10.35.116.2 8c13abcf-d106-4b6c-99a6-f2d0c0162f64 False < /var/cache/ovirt-engine/vdsm-bootstrap-2.tar. (Stage: Running first installation script on Host) >2012-09-02 14:20:38,384 INFO [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-4-thread-50) SSH execute 10.35.116.2 'umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar -C "${MYTMP}" --no-same-permissions -o -x && "${MYTMP}"/setup -c 'ssl=true;management_port=54321' -O 'qa.lab.tlv.redhat.com' -t 2012-09-02T11:20:38 -f /tmp/firewall.conf.8c13abcf-d106-4b6c-99a6-f2d0c0162f64 -S /tmp/ovirt-id_rsa_8c13abcf-d106-4b6c-99a6-f2d0c0162f64 -p 80 -b http://gadi-rhevm.qa.lab.tlv.redhat.com:80/Components/vds/ http://gadi-rhevm.qa.lab.tlv.redhat.com:80/Components/vds/ 10.35.116.2 8c13abcf-d106-4b6c-99a6-f2d0c0162f64 False' >2012-09-02 14:20:38,510 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-7) Installation of 10.35.116.2. Received message: <BSTRAP component='INSTALLER' status='OK' message='Test platform succeeded'/>. FYI. (Stage: Running first installation script on Host) >2012-09-02 14:20:39,611 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-7) Installation of 10.35.116.2. Received message: <BSTRAP component='SET_SYSTEM_TIME' status='OK' message='setSystemTime ended successfully'/>. FYI. (Stage: Running first installation script on Host) >2012-09-02 14:20:39,660 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-4) Initializing Host: camel-vdsb.qa.lab.tlv.redhat.com >2012-09-02 14:20:41,747 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-7) Installation of 10.35.116.2. Received message: <BSTRAP component='RHN_REGISTRATION' status='OK' message='Host properly registered with RHN/Satellite.'/>. FYI. (Stage: Running first installation script on Host) >2012-09-02 14:20:41,952 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-7) Installation of 10.35.116.2. Received message: <BSTRAP component='VDSM_MAJOR_VER' status='OK' message='Available VDSM matches requirements'/>. FYI. (Stage: Running first installation script on Host) >2012-09-02 14:20:41,962 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-7) Installation of 10.35.116.2. Received message: <BSTRAP component='VT_SVM' status='OK' processor='Intel' message='Server supports virtualization'/>. FYI. (Stage: Running first installation script on Host) >2012-09-02 14:20:41,973 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-7) Installation of 10.35.116.2. Received message: <BSTRAP component='OS' status='OK' type='RHEL6' message='Supported platform version'/>. FYI. (Stage: Running first installation script on Host) >2012-09-02 14:20:41,990 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-7) Installation of 10.35.116.2. Received message: <BSTRAP component='KERNEL' status='OK' version='279' message='Supported kernel version: 279'/>. FYI. (Stage: Running first installation script on Host) >2012-09-02 14:20:42,096 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-7) Installation of 10.35.116.2. Received message: <BSTRAP component='REQ PACKAGES' status='OK' result='SDL.x86_64' message='SDL-1.2.14-3.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-09-02 14:20:42,124 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-7) Installation of 10.35.116.2. Received message: <BSTRAP component='REQ PACKAGES' status='OK' result='bridge-utils.x86_64' message='bridge-utils-1.2-9.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-09-02 14:20:42,143 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-7) Installation of 10.35.116.2. Received message: <BSTRAP component='REQ PACKAGES' status='OK' result='mesa-libGLU.x86_64' message='mesa-libGLU-7.11-5.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-09-02 14:20:42,163 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-7) Installation of 10.35.116.2. Received message: <BSTRAP component='REQ PACKAGES' status='OK' result='openssl.x86_64' message='openssl-1.0.0-25.el6_3.1.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-09-02 14:20:42,185 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-7) Installation of 10.35.116.2. Received message: <BSTRAP component='REQ PACKAGES' status='OK' result='m2crypto.x86_64' message='m2crypto-0.20.2-9.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-09-02 14:20:42,205 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-7) Installation of 10.35.116.2. Received message: <BSTRAP component='REQ PACKAGES' status='OK' result='rsync.x86_64' message='rsync-3.0.6-9.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-09-02 14:20:42,221 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-7) Installation of 10.35.116.2. Received message: <BSTRAP component='VDS PACKAGES' status='WARN' result='qemu-kvm' message='package qemu-kvm is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-09-02 14:20:42,246 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-7) Installation of 10.35.116.2. Received message: <BSTRAP component='VDS PACKAGES' status='WARN' result='qemu-kvm-tools' message='package qemu-kvm-tools is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-09-02 14:20:42,258 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-7) Installation of 10.35.116.2. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='vdsm' message='vdsm-4.9.6-31.0.el6_3.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-09-02 14:20:42,285 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-7) Installation of 10.35.116.2. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='vdsm-cli' message='vdsm-cli-4.9.6-31.0.el6_3.noarch '/>. FYI. (Stage: Running first installation script on Host) >2012-09-02 14:20:42,310 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-7) Installation of 10.35.116.2. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='libjpeg' message='libjpeg-6b-46.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-09-02 14:20:42,338 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-7) Installation of 10.35.116.2. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='spice-server' message='spice-server-0.10.1-10.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-09-02 14:20:42,346 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-7) Installation of 10.35.116.2. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='pixman' message='pixman-0.18.4-1.el6_0.1.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-09-02 14:20:42,364 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-7) Installation of 10.35.116.2. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='seabios' message='seabios-0.6.1.2-19.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-09-02 14:20:42,372 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-7) Installation of 10.35.116.2. Received message: <BSTRAP component='VDS PACKAGES' status='WARN' result='qemu-img' message='package qemu-img is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-09-02 14:20:42,392 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-7) Installation of 10.35.116.2. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='fence-agents' message='fence-agents-3.1.5-17.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-09-02 14:20:42,412 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-7) Installation of 10.35.116.2. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='libselinux-python' message='libselinux-python-2.0.94-5.3.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-09-02 14:20:42,430 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-7) Installation of 10.35.116.2. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='sanlock' message='sanlock-2.3-3.el6_3.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-09-02 14:20:42,449 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-7) Installation of 10.35.116.2. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='sanlock-python' message='sanlock-python-2.3-3.el6_3.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-09-02 14:21:59,868 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-7) Installation of 10.35.116.2. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='vdsm' message='vdsm-4.9.6-31.0.el6_3.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-09-02 14:22:03,785 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-7) Installation of 10.35.116.2. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='vdsm-cli' message='vdsm-cli-4.9.6-31.0.el6_3.noarch '/>. FYI. (Stage: Running first installation script on Host) >2012-09-02 14:22:07,136 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-7) Installation of 10.35.116.2. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='libjpeg' message='libjpeg-6b-46.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-09-02 14:22:09,849 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-7) Installation of 10.35.116.2. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='spice-server' message='spice-server-0.10.1-10.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-09-02 14:22:12,304 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-7) Installation of 10.35.116.2. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='pixman' message='pixman-0.18.4-1.el6_0.1.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-09-02 14:22:14,791 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-7) Installation of 10.35.116.2. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='seabios' message='seabios-0.6.1.2-19.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-09-02 14:22:17,373 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-7) Installation of 10.35.116.2. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='fence-agents' message='fence-agents-3.1.5-17.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-09-02 14:22:20,063 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-7) Installation of 10.35.116.2. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='libselinux-python' message='libselinux-python-2.0.94-5.3.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-09-02 14:22:23,732 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-7) Installation of 10.35.116.2. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='sanlock' message='sanlock-2.3-3.el6_3.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-09-02 14:22:26,499 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-7) Installation of 10.35.116.2. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='sanlock-python' message='sanlock-python-2.3-3.el6_3.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-09-02 14:22:29,251 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-7) Installation of 10.35.116.2. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='qemu-img' message='package qemu-img is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-09-02 14:22:31,810 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-7) Installation of 10.35.116.2. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='qemu-kvm-tools' message='package qemu-kvm-tools is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-09-02 14:22:34,386 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-7) Installation of 10.35.116.2. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='qemu-kvm' message='package qemu-kvm is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-09-02 14:22:35,379 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-7) Installation of 10.35.116.2. Received message: <BSTRAP component='CreateConf' status='OK' message='Basic configuration found, skipping this step'/>. FYI. (Stage: Running first installation script on Host) >2012-09-02 14:22:35,397 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-7) Installation of 10.35.116.2. Received message: <BSTRAP component='CheckLocalHostname' status='OK' message='Local hostname is correct.'/>. FYI. (Stage: Running first installation script on Host) >2012-09-02 14:22:35,431 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-7) Installation of 10.35.116.2. Received message: <BSTRAP component='SetNetworking' status='OK' message='Bridge management already exists. Skipping bridge creation.'/>. FYI. (Stage: Running first installation script on Host) >2012-09-02 14:22:35,467 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-7) Installation of 10.35.116.2. Received message: <BSTRAP component='SetSSHAccess' status='OK' message='SUCCESS'/>. FYI. (Stage: Running first installation script on Host) >2012-09-02 14:22:35,484 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-7) Installation of 10.35.116.2. Received message: <BSTRAP component='Firewall' status='OK' message='overridden firewall successfully'/>. FYI. (Stage: Running first installation script on Host) >2012-09-02 14:22:35,503 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-7) Installation of 10.35.116.2. Received message: <BSTRAP component='VerifyServices' status='OK' message='Needed services set'/>. FYI. (Stage: Running first installation script on Host) >2012-09-02 14:22:43,833 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-7) Installation of 10.35.116.2. Received message: <BSTRAP component='Encryption setup' status='OK' message='Ended successfully'/>. FYI. (Stage: Running first installation script on Host) >2012-09-02 14:22:43,892 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-7) Installation of 10.35.116.2. Received message: <BSTRAP component='RHEV_INSTALL' status='OK'/>. Stage completed. (Stage: Running first installation script on Host) >2012-09-02 14:22:43,910 ERROR [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-50) [6daa7da2] Installation of 10.35.116.2. Error: Repository RHEVM-Build is listed more than once in the configuration >Repository KIPI is listed more than once in the configuration >Repository JBOSS is listed more than once in the configuration >Repository Missing-Package is listed more than once in the configuration >Repository TLV is listed more than once in the configuration >Repository rhel-source is listed more than once in the configuration >Repository rhel-source-beta is listed more than once in the configuration >Repository RHEVM-Build is listed more than once in the configuration >Repository KIPI is listed more than once in the configuration >Repository JBOSS is listed more than once in the configuration >Repository Missing-Package is listed more than once in the configuration >Repository TLV is listed more than once in the configuration >Repository rhel-source is listed more than once in the configuration >Repository rhel-source-beta is listed more than once in the configuration >. (Stage: Downloading certificate request from Host) >2012-09-02 14:22:43,910 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-50) [6daa7da2] VDS message: Repository RHEVM-Build is listed more than once in the configuration >2012-09-02 14:22:43,910 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-50) [6daa7da2] VDS message: Repository KIPI is listed more than once in the configuration >2012-09-02 14:22:43,910 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-50) [6daa7da2] VDS message: Repository JBOSS is listed more than once in the configuration >2012-09-02 14:22:43,910 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-50) [6daa7da2] VDS message: Repository Missing-Package is listed more than once in the configuration >2012-09-02 14:22:43,910 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-50) [6daa7da2] VDS message: Repository TLV is listed more than once in the configuration >2012-09-02 14:22:43,910 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-50) [6daa7da2] VDS message: Repository rhel-source is listed more than once in the configuration >2012-09-02 14:22:43,910 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-50) [6daa7da2] VDS message: Repository rhel-source-beta is listed more than once in the configuration >2012-09-02 14:22:43,910 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-50) [6daa7da2] VDS message: Repository RHEVM-Build is listed more than once in the configuration >2012-09-02 14:22:43,910 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-50) [6daa7da2] VDS message: Repository KIPI is listed more than once in the configuration >2012-09-02 14:22:43,910 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-50) [6daa7da2] VDS message: Repository JBOSS is listed more than once in the configuration >2012-09-02 14:22:43,910 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-50) [6daa7da2] VDS message: Repository Missing-Package is listed more than once in the configuration >2012-09-02 14:22:43,910 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-50) [6daa7da2] VDS message: Repository TLV is listed more than once in the configuration >2012-09-02 14:22:43,910 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-50) [6daa7da2] VDS message: Repository rhel-source is listed more than once in the configuration >2012-09-02 14:22:43,911 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-50) [6daa7da2] VDS message: Repository rhel-source-beta is listed more than once in the configuration >2012-09-02 14:22:43,911 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-50) [6daa7da2] Script ended, result is {1} >2012-09-02 14:22:43,911 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-50) [6daa7da2] Installation of 10.35.116.2. Executing installation stage. (Stage: Downloading certificate request from Host) >2012-09-02 14:22:43,911 INFO [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-4-thread-50) SSH receive 10.35.116.2:'/tmp/cert_8c13abcf-d106-4b6c-99a6-f2d0c0162f64.req' '/etc/pki/ovirt-engine/requests/cert_8c13abcf-d106-4b6c-99a6-f2d0c0162f64.req') >2012-09-02 14:22:43,999 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-50) [6daa7da2] Installation of 10.35.116.2. successfully done sftp operation ( Stage: Downloading certificate request from Host) >2012-09-02 14:22:43,999 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-50) [6daa7da2] DownloadCertificateRequest ended:true >2012-09-02 14:22:43,999 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-50) [6daa7da2] Installation of 10.35.116.2. Executing installation stage. (Stage: Sign certificate request and generate certificate) >2012-09-02 14:22:45,006 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-50) [6daa7da2] SignCertificateRequest ended:true >2012-09-02 14:22:45,006 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-50) [6daa7da2] Installation of 10.35.116.2. Executing installation stage. (Stage: Upload signed sertificate to Host) >2012-09-02 14:22:45,040 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-50) [6daa7da2] Installation of 10.35.116.2. successfully done sftp operation ( Stage: Upload signed sertificate to Host) >2012-09-02 14:22:45,040 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-50) [6daa7da2] UploadSignedCertificate ended:true >2012-09-02 14:22:45,040 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-50) [6daa7da2] Installation of 10.35.116.2. Executing installation stage. (Stage: Upload Cerficate Autority to Host) >2012-09-02 14:22:45,102 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-50) [6daa7da2] Installation of 10.35.116.2. successfully done sftp operation ( Stage: Upload Cerficate Autority to Host) >2012-09-02 14:22:45,102 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-50) [6daa7da2] Installation of 10.35.116.2. Executing installation stage. (Stage: Running second installation script on Host) >2012-09-02 14:22:45,103 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-50) [6daa7da2] Installation of 10.35.116.2. Sending SSH Command umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar -C "${MYTMP}" --no-same-permissions -o -x && "${MYTMP}"/setup -c 'ssl=true;management_port=54321' -O 'qa.lab.tlv.redhat.com' -t 2012-09-02T11:20:38 -f /tmp/firewall.conf.8c13abcf-d106-4b6c-99a6-f2d0c0162f64 -S /tmp/ovirt-id_rsa_8c13abcf-d106-4b6c-99a6-f2d0c0162f64 -p 80 -b http://gadi-rhevm.qa.lab.tlv.redhat.com:80/Components/vds/ http://gadi-rhevm.qa.lab.tlv.redhat.com:80/Components/vds/ 10.35.116.2 8c13abcf-d106-4b6c-99a6-f2d0c0162f64 True < /var/cache/ovirt-engine/vdsm-bootstrap-2.tar. (Stage: Running second installation script on Host) >2012-09-02 14:22:45,103 INFO [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-4-thread-50) SSH execute 10.35.116.2 'umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar -C "${MYTMP}" --no-same-permissions -o -x && "${MYTMP}"/setup -c 'ssl=true;management_port=54321' -O 'qa.lab.tlv.redhat.com' -t 2012-09-02T11:20:38 -f /tmp/firewall.conf.8c13abcf-d106-4b6c-99a6-f2d0c0162f64 -S /tmp/ovirt-id_rsa_8c13abcf-d106-4b6c-99a6-f2d0c0162f64 -p 80 -b http://gadi-rhevm.qa.lab.tlv.redhat.com:80/Components/vds/ http://gadi-rhevm.qa.lab.tlv.redhat.com:80/Components/vds/ 10.35.116.2 8c13abcf-d106-4b6c-99a6-f2d0c0162f64 True' >2012-09-02 14:22:45,211 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-7) Installation of 10.35.116.2. Received message: <BSTRAP component='INSTALLER' status='OK' message='Test platform succeeded'/>. FYI. (Stage: Running second installation script on Host) >2012-09-02 14:23:03,750 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8009-8) Checking if user admin@internal is an admin, result true >2012-09-02 14:23:03,751 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8009-8) Running command: LoginUserCommand internal: false. >2012-09-02 14:23:03,810 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8009-8) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2012-09-02 14:23:04,067 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8009-6) Checking if user admin@internal is an admin, result true >2012-09-02 14:23:04,067 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8009-6) Running command: LoginUserCommand internal: false. >2012-09-02 14:23:04,125 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8009-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2012-09-02 14:23:06,487 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DiscoverSendTargetsVDSCommand] (ajp-/127.0.0.1:8009-4) START, DiscoverSendTargetsVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, connection=org.ovirt.engine.core.common.businessentities.storage_server_connections@0), log id: 6fe68a50 >2012-09-02 14:23:06,740 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DiscoverSendTargetsVDSCommand] (ajp-/127.0.0.1:8009-4) FINISH, DiscoverSendTargetsVDSCommand, return: [org.ovirt.engine.core.common.businessentities.storage_server_connections@0, org.ovirt.engine.core.common.businessentities.storage_server_connections@0, org.ovirt.engine.core.common.businessentities.storage_server_connections@0, org.ovirt.engine.core.common.businessentities.storage_server_connections@0, org.ovirt.engine.core.common.businessentities.storage_server_connections@0, org.ovirt.engine.core.common.businessentities.storage_server_connections@0, org.ovirt.engine.core.common.businessentities.storage_server_connections@0, org.ovirt.engine.core.common.businessentities.storage_server_connections@0, org.ovirt.engine.core.common.businessentities.storage_server_connections@0, org.ovirt.engine.core.common.businessentities.storage_server_connections@0, org.ovirt.engine.core.common.businessentities.storage_server_connections@0, org.ovirt.engine.core.common.businessentities.storage_server_connections@0, org.ovirt.engine.core.common.businessentities.storage_server_connections@0, org.ovirt.engine.core.common.businessentities.storage_server_connections@0, org.ovirt.engine.core.common.businessentities.storage_server_connections@0, org.ovirt.engine.core.common.businessentities.storage_server_connections@0, org.ovirt.engine.core.common.businessentities.storage_server_connections@0, org.ovirt.engine.core.common.businessentities.storage_server_connections@0, org.ovirt.engine.core.common.businessentities.storage_server_connections@0, org.ovirt.engine.core.common.businessentities.storage_server_connections@0, org.ovirt.engine.core.common.businessentities.storage_server_connections@0, org.ovirt.engine.core.common.businessentities.storage_server_connections@0, org.ovirt.engine.core.common.businessentities.storage_server_connections@0, org.ovirt.engine.core.common.businessentities.storage_server_connections@0, org.ovirt.engine.core.common.businessentities.storage_server_connections@0, org.ovirt.engine.core.common.businessentities.storage_server_connections@0, org.ovirt.engine.core.common.businessentities.storage_server_connections@0, org.ovirt.engine.core.common.businessentities.storage_server_connections@0, org.ovirt.engine.core.common.businessentities.storage_server_connections@0, org.ovirt.engine.core.common.businessentities.storage_server_connections@0, org.ovirt.engine.core.common.businessentities.storage_server_connections@0, org.ovirt.engine.core.common.businessentities.storage_server_connections@0, org.ovirt.engine.core.common.businessentities.storage_server_connections@0, org.ovirt.engine.core.common.businessentities.storage_server_connections@0, org.ovirt.engine.core.common.businessentities.storage_server_connections@0, org.ovirt.engine.core.common.businessentities.storage_server_connections@0, org.ovirt.engine.core.common.businessentities.storage_server_connections@0, org.ovirt.engine.core.common.businessentities.storage_server_connections@0, org.ovirt.engine.core.common.businessentities.storage_server_connections@0, org.ovirt.engine.core.common.businessentities.storage_server_connections@0, org.ovirt.engine.core.common.businessentities.storage_server_connections@0, org.ovirt.engine.core.common.businessentities.storage_server_connections@0, org.ovirt.engine.core.common.businessentities.storage_server_connections@0, org.ovirt.engine.core.common.businessentities.storage_server_connections@0, org.ovirt.engine.core.common.businessentities.storage_server_connections@0, org.ovirt.engine.core.common.businessentities.storage_server_connections@0, org.ovirt.engine.core.common.businessentities.storage_server_connections@0, org.ovirt.engine.core.common.businessentities.storage_server_connections@0, org.ovirt.engine.core.common.businessentities.storage_server_connections@0, org.ovirt.engine.core.common.businessentities.storage_server_connections@0, org.ovirt.engine.core.common.businessentities.storage_server_connections@0], log id: 6fe68a50 >2012-09-02 14:23:06,950 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (ajp-/127.0.0.1:8009-9) [6a87d94a] START, ValidateStorageServerConnectionVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = ISCSI, connectionList = [{ id: null, connection: orion.qa.lab.tlv.redhat.com };]), log id: 733816e >2012-09-02 14:23:06,977 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (ajp-/127.0.0.1:8009-9) [6a87d94a] FINISH, ValidateStorageServerConnectionVDSCommand, return: {00000000-0000-0000-0000-000000000000=0}, log id: 733816e >2012-09-02 14:23:06,998 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (ajp-/127.0.0.1:8009-9) [6a87d94a] Running command: ConnectStorageToVdsCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2012-09-02 14:23:06,999 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8009-9) [6a87d94a] START, ConnectStorageServerVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = ISCSI, connectionList = [{ id: null, connection: orion.qa.lab.tlv.redhat.com };]), log id: 472b02b7 >2012-09-02 14:23:07,689 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8009-9) [6a87d94a] FINISH, ConnectStorageServerVDSCommand, return: {00000000-0000-0000-0000-000000000000=0}, log id: 472b02b7 >2012-09-02 14:23:10,460 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (ajp-/127.0.0.1:8009-3) [4357f1eb] START, ValidateStorageServerConnectionVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = ISCSI, connectionList = [{ id: null, connection: 10.35.64.10 };]), log id: 612e3937 >2012-09-02 14:23:10,501 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (ajp-/127.0.0.1:8009-3) [4357f1eb] FINISH, ValidateStorageServerConnectionVDSCommand, return: {00000000-0000-0000-0000-000000000000=0}, log id: 612e3937 >2012-09-02 14:23:10,542 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (ajp-/127.0.0.1:8009-3) [4357f1eb] Running command: ConnectStorageToVdsCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2012-09-02 14:23:10,544 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8009-3) [4357f1eb] START, ConnectStorageServerVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = ISCSI, connectionList = [{ id: null, connection: 10.35.64.10 };]), log id: 3966b24c >2012-09-02 14:23:10,642 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8009-3) [4357f1eb] FINISH, ConnectStorageServerVDSCommand, return: {00000000-0000-0000-0000-000000000000=0}, log id: 3966b24c >2012-09-02 14:23:10,662 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetDeviceListVDSCommand] (ajp-/127.0.0.1:8009-3) START, GetDeviceListVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storageType=ISCSI), log id: 73415f74 >2012-09-02 14:23:12,048 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetDeviceListVDSCommand] (ajp-/127.0.0.1:8009-3) FINISH, GetDeviceListVDSCommand, return: [org.ovirt.engine.core.common.businessentities.LUNs@fb04e8c9, org.ovirt.engine.core.common.businessentities.LUNs@602c4909], log id: 73415f74 >2012-09-02 14:23:12,139 INFO [org.ovirt.engine.core.bll.storage.AddSANStorageDomainCommand] (ajp-/127.0.0.1:8009-3) [2b3799f4] Running command: AddSANStorageDomainCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2012-09-02 14:23:12,177 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVGVDSCommand] (ajp-/127.0.0.1:8009-3) [2b3799f4] START, CreateVGVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storageDomainId=2e857431-edc6-4a93-8d89-8b9a405dba47, deviceList=[1gickowic-lun113459821], force=false), log id: 24088e74 >2012-09-02 14:23:12,380 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (ajp-/127.0.0.1:8009-3) [2b3799f4] Failed in CreateVGVDS method >2012-09-02 14:23:12,381 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (ajp-/127.0.0.1:8009-3) [2b3799f4] Error code PhysDevInitializationError and error message VDSGenericException: VDSErrorException: Failed to CreateVGVDS, error = Failed to initialize physical device: ("['/dev/mapper/1gickowic-lun113459821']",) >2012-09-02 14:23:12,383 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (ajp-/127.0.0.1:8009-3) [2b3799f4] Command org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVGVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.irsbroker.OneUuidReturnForXmlRpc >mUuid Null >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 601 >mMessage Failed to initialize physical device: ("['/dev/mapper/1gickowic-lun113459821']",) > > >2012-09-02 14:23:12,383 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (ajp-/127.0.0.1:8009-3) [2b3799f4] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 14:23:12,391 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (ajp-/127.0.0.1:8009-3) [2b3799f4] Command CreateVGVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to CreateVGVDS, error = Failed to initialize physical device: ("['/dev/mapper/1gickowic-lun113459821']",) >2012-09-02 14:23:12,391 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVGVDSCommand] (ajp-/127.0.0.1:8009-3) [2b3799f4] FINISH, CreateVGVDSCommand, log id: 24088e74 >2012-09-02 14:23:12,394 ERROR [org.ovirt.engine.core.bll.storage.AddSANStorageDomainCommand] (ajp-/127.0.0.1:8009-3) [2b3799f4] Command org.ovirt.engine.core.bll.storage.AddSANStorageDomainCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to CreateVGVDS, error = Failed to initialize physical device: ("['/dev/mapper/1gickowic-lun113459821']",) >2012-09-02 14:23:12,418 INFO [org.ovirt.engine.core.bll.storage.AddSANStorageDomainCommand] (ajp-/127.0.0.1:8009-3) [2b3799f4] Command [id=ded1b0cd-4e92-4306-a912-04727abed961]: Compensating NEW_ENTITY_ID of org.ovirt.engine.core.common.businessentities.storage_domain_dynamic; snapshot: 2e857431-edc6-4a93-8d89-8b9a405dba47. >2012-09-02 14:23:12,425 INFO [org.ovirt.engine.core.bll.storage.AddSANStorageDomainCommand] (ajp-/127.0.0.1:8009-3) [2b3799f4] Command [id=ded1b0cd-4e92-4306-a912-04727abed961]: Compensating NEW_ENTITY_ID of org.ovirt.engine.core.common.businessentities.storage_domain_static; snapshot: 2e857431-edc6-4a93-8d89-8b9a405dba47. >2012-09-02 14:23:12,454 ERROR [org.ovirt.engine.core.bll.storage.AddSANStorageDomainCommand] (ajp-/127.0.0.1:8009-3) [2b3799f4] Transaction rolled-back for command: org.ovirt.engine.core.bll.storage.AddSANStorageDomainCommand. >2012-09-02 14:23:12,496 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (ajp-/127.0.0.1:8009-3) Operation Failed: [] >2012-09-02 14:23:17,934 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-7) Installation of 10.35.116.2. Received message: <BSTRAP component='instCert' status='OK'/>. FYI. (Stage: Running second installation script on Host) >2012-09-02 14:23:17,966 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-7) Installation of 10.35.116.2. Received message: <BSTRAP component='CoreDump' status='OK'/>. FYI. (Stage: Running second installation script on Host) >2012-09-02 14:23:17,982 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-7) Installation of 10.35.116.2. Received message: <BSTRAP component='cleanAll' status='OK'/>. FYI. (Stage: Running second installation script on Host) >2012-09-02 14:23:17,997 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-7) Installation of 10.35.116.2. Received message: <BSTRAP component='VDS Configuration' status='OK'/>. FYI. (Stage: Running second installation script on Host) >2012-09-02 14:23:18,011 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-7) Installation of 10.35.116.2. Received message: <BSTRAP component='RHEV_INSTALL' status='OK'/>. Stage completed. (Stage: Running second installation script on Host) >2012-09-02 14:23:18,025 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-7) Installation of 10.35.116.2. Received message: <BSTRAP component='Reboot' status='OK' message='Rebooting machine' />. FYI. (Stage: Host installation complete) >2012-09-02 14:23:18,039 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-50) [6daa7da2] Script ended, result is {1} >2012-09-02 14:23:18,041 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-4-thread-50) [6daa7da2] After Installation pool-4-thread-50 >2012-09-02 14:23:18,042 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-50) [6daa7da2] START, SetVdsStatusVDSCommand(vdsId = 3880c572-f4f0-11e1-9a52-001a4a169705, status=Reboot, nonOperationalReason=NONE), log id: 42402416 >2012-09-02 14:23:18,076 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-50) [6daa7da2] FINISH, SetVdsStatusVDSCommand, log id: 42402416 >2012-09-02 14:23:18,076 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-4-thread-48) [6daa7da2] Waiting 300 seconds, for server to finish reboot process. >2012-09-02 14:24:18,455 INFO [org.ovirt.engine.core.bll.LoginAdminUserCommand] (ajp-/127.0.0.1:8009-8) Checking if user admin@internal is an admin, result true >2012-09-02 14:24:18,455 INFO [org.ovirt.engine.core.bll.LoginAdminUserCommand] (ajp-/127.0.0.1:8009-8) Running command: LoginAdminUserCommand internal: false. >2012-09-02 14:24:19,316 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8009-1) calling GetConfigurationValueQuery (CustomPublicConfig_AppsWebSite) with null version, using default general for version >2012-09-02 14:25:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-27) Autorecovering hosts is disabled, skipping >2012-09-02 14:25:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-27) Autorecovering storage domains is disabled, skipping >2012-09-02 14:27:32,136 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8009-4) Checking if user admin@internal is an admin, result true >2012-09-02 14:27:32,137 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8009-4) Running command: LoginUserCommand internal: false. >2012-09-02 14:27:32,243 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8009-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2012-09-02 14:27:32,444 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8009-3) Checking if user admin@internal is an admin, result true >2012-09-02 14:27:32,444 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8009-3) Running command: LoginUserCommand internal: false. >2012-09-02 14:27:32,499 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8009-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2012-09-02 14:27:34,127 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DiscoverSendTargetsVDSCommand] (ajp-/127.0.0.1:8009-9) START, DiscoverSendTargetsVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, connection=org.ovirt.engine.core.common.businessentities.storage_server_connections@0), log id: ccda189 >2012-09-02 14:27:34,320 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DiscoverSendTargetsVDSCommand] (ajp-/127.0.0.1:8009-9) FINISH, DiscoverSendTargetsVDSCommand, return: [org.ovirt.engine.core.common.businessentities.storage_server_connections@0, org.ovirt.engine.core.common.businessentities.storage_server_connections@0, org.ovirt.engine.core.common.businessentities.storage_server_connections@0, org.ovirt.engine.core.common.businessentities.storage_server_connections@0, org.ovirt.engine.core.common.businessentities.storage_server_connections@0, org.ovirt.engine.core.common.businessentities.storage_server_connections@0, org.ovirt.engine.core.common.businessentities.storage_server_connections@0, org.ovirt.engine.core.common.businessentities.storage_server_connections@0, org.ovirt.engine.core.common.businessentities.storage_server_connections@0, org.ovirt.engine.core.common.businessentities.storage_server_connections@0, org.ovirt.engine.core.common.businessentities.storage_server_connections@0, org.ovirt.engine.core.common.businessentities.storage_server_connections@0, org.ovirt.engine.core.common.businessentities.storage_server_connections@0, org.ovirt.engine.core.common.businessentities.storage_server_connections@0, org.ovirt.engine.core.common.businessentities.storage_server_connections@0, org.ovirt.engine.core.common.businessentities.storage_server_connections@0, org.ovirt.engine.core.common.businessentities.storage_server_connections@0, org.ovirt.engine.core.common.businessentities.storage_server_connections@0, org.ovirt.engine.core.common.businessentities.storage_server_connections@0, org.ovirt.engine.core.common.businessentities.storage_server_connections@0, org.ovirt.engine.core.common.businessentities.storage_server_connections@0, org.ovirt.engine.core.common.businessentities.storage_server_connections@0, org.ovirt.engine.core.common.businessentities.storage_server_connections@0, org.ovirt.engine.core.common.businessentities.storage_server_connections@0, org.ovirt.engine.core.common.businessentities.storage_server_connections@0, org.ovirt.engine.core.common.businessentities.storage_server_connections@0, org.ovirt.engine.core.common.businessentities.storage_server_connections@0, org.ovirt.engine.core.common.businessentities.storage_server_connections@0, org.ovirt.engine.core.common.businessentities.storage_server_connections@0, org.ovirt.engine.core.common.businessentities.storage_server_connections@0, org.ovirt.engine.core.common.businessentities.storage_server_connections@0, org.ovirt.engine.core.common.businessentities.storage_server_connections@0, org.ovirt.engine.core.common.businessentities.storage_server_connections@0, org.ovirt.engine.core.common.businessentities.storage_server_connections@0, org.ovirt.engine.core.common.businessentities.storage_server_connections@0, org.ovirt.engine.core.common.businessentities.storage_server_connections@0, org.ovirt.engine.core.common.businessentities.storage_server_connections@0, org.ovirt.engine.core.common.businessentities.storage_server_connections@0, org.ovirt.engine.core.common.businessentities.storage_server_connections@0, org.ovirt.engine.core.common.businessentities.storage_server_connections@0, org.ovirt.engine.core.common.businessentities.storage_server_connections@0, org.ovirt.engine.core.common.businessentities.storage_server_connections@0, org.ovirt.engine.core.common.businessentities.storage_server_connections@0, org.ovirt.engine.core.common.businessentities.storage_server_connections@0, org.ovirt.engine.core.common.businessentities.storage_server_connections@0, org.ovirt.engine.core.common.businessentities.storage_server_connections@0, org.ovirt.engine.core.common.businessentities.storage_server_connections@0, org.ovirt.engine.core.common.businessentities.storage_server_connections@0, org.ovirt.engine.core.common.businessentities.storage_server_connections@0, org.ovirt.engine.core.common.businessentities.storage_server_connections@0, org.ovirt.engine.core.common.businessentities.storage_server_connections@0], log id: ccda189 >2012-09-02 14:27:34,516 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (ajp-/127.0.0.1:8009-4) [309b3d67] START, ValidateStorageServerConnectionVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = ISCSI, connectionList = [{ id: null, connection: orion.qa.lab.tlv.redhat.com };]), log id: 7f94e0f0 >2012-09-02 14:27:34,537 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (ajp-/127.0.0.1:8009-4) [309b3d67] FINISH, ValidateStorageServerConnectionVDSCommand, return: {00000000-0000-0000-0000-000000000000=0}, log id: 7f94e0f0 >2012-09-02 14:27:34,559 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (ajp-/127.0.0.1:8009-4) [309b3d67] Running command: ConnectStorageToVdsCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2012-09-02 14:27:34,560 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8009-4) [309b3d67] START, ConnectStorageServerVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = ISCSI, connectionList = [{ id: null, connection: orion.qa.lab.tlv.redhat.com };]), log id: 1ea55f0f >2012-09-02 14:27:35,211 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8009-4) [309b3d67] FINISH, ConnectStorageServerVDSCommand, return: {00000000-0000-0000-0000-000000000000=0}, log id: 1ea55f0f >2012-09-02 14:27:35,584 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (ajp-/127.0.0.1:8009-6) [3fd08b99] START, ValidateStorageServerConnectionVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = ISCSI, connectionList = [{ id: null, connection: 10.35.64.10 };]), log id: 576ccd49 >2012-09-02 14:27:35,620 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (ajp-/127.0.0.1:8009-6) [3fd08b99] FINISH, ValidateStorageServerConnectionVDSCommand, return: {00000000-0000-0000-0000-000000000000=0}, log id: 576ccd49 >2012-09-02 14:27:35,655 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (ajp-/127.0.0.1:8009-6) [3fd08b99] Running command: ConnectStorageToVdsCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2012-09-02 14:27:35,657 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8009-6) [3fd08b99] START, ConnectStorageServerVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = ISCSI, connectionList = [{ id: null, connection: 10.35.64.10 };]), log id: 62932891 >2012-09-02 14:27:35,738 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8009-6) [3fd08b99] FINISH, ConnectStorageServerVDSCommand, return: {00000000-0000-0000-0000-000000000000=0}, log id: 62932891 >2012-09-02 14:27:35,750 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetDeviceListVDSCommand] (ajp-/127.0.0.1:8009-6) START, GetDeviceListVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storageType=ISCSI), log id: 64363c33 >2012-09-02 14:27:36,648 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetDeviceListVDSCommand] (ajp-/127.0.0.1:8009-6) FINISH, GetDeviceListVDSCommand, return: [org.ovirt.engine.core.common.businessentities.LUNs@9f3402aa, org.ovirt.engine.core.common.businessentities.LUNs@602c4909], log id: 64363c33 >2012-09-02 14:27:36,737 INFO [org.ovirt.engine.core.bll.storage.AddSANStorageDomainCommand] (ajp-/127.0.0.1:8009-6) [2097bc2a] Running command: AddSANStorageDomainCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2012-09-02 14:27:36,760 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVGVDSCommand] (ajp-/127.0.0.1:8009-6) [2097bc2a] START, CreateVGVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storageDomainId=5df9c9a3-d1d1-452c-9e01-4904708090dd, deviceList=[1gickowic-lun113459821], force=false), log id: 3b69ccf3 >2012-09-02 14:27:37,567 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVGVDSCommand] (ajp-/127.0.0.1:8009-6) [2097bc2a] FINISH, CreateVGVDSCommand, return: wHxJnu-MPiI-woQw-wlPj-W981-fCCv-4srSLx, log id: 3b69ccf3 >2012-09-02 14:27:37,586 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (ajp-/127.0.0.1:8009-6) [2097bc2a] START, CreateStorageDomainVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storageDomain=org.ovirt.engine.core.common.businessentities.storage_domain_static@5e2eb592, args=wHxJnu-MPiI-woQw-wlPj-W981-fCCv-4srSLx), log id: 5a6a057 >2012-09-02 14:27:43,833 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (ajp-/127.0.0.1:8009-6) [2097bc2a] FINISH, CreateStorageDomainVDSCommand, log id: 5a6a057 >2012-09-02 14:27:43,838 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStorageDomainStatsVDSCommand] (ajp-/127.0.0.1:8009-6) [2097bc2a] START, GetStorageDomainStatsVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storageDomainId=5df9c9a3-d1d1-452c-9e01-4904708090dd), log id: 5fb848cb >2012-09-02 14:27:44,038 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStorageDomainStatsVDSCommand] (ajp-/127.0.0.1:8009-6) [2097bc2a] FINISH, GetStorageDomainStatsVDSCommand, return: org.ovirt.engine.core.common.businessentities.storage_domains@78f31f39, log id: 5fb848cb >2012-09-02 14:27:44,076 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVGInfoVDSCommand] (ajp-/127.0.0.1:8009-6) [2097bc2a] START, GetVGInfoVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, VGID=wHxJnu-MPiI-woQw-wlPj-W981-fCCv-4srSLx), log id: 1fabfc9d >2012-09-02 14:27:44,114 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVGInfoVDSCommand] (ajp-/127.0.0.1:8009-6) [2097bc2a] FINISH, GetVGInfoVDSCommand, return: [org.ovirt.engine.core.common.businessentities.LUNs@1efac2df], log id: 1fabfc9d >2012-09-02 14:27:44,956 INFO [org.ovirt.engine.core.bll.storage.AttachStorageDomainToPoolCommand] (ajp-/127.0.0.1:8009-6) [702e1aa2] Running command: AttachStorageDomainToPoolCommand internal: false. Entities affected : ID: 5df9c9a3-d1d1-452c-9e01-4904708090dd Type: Storage >2012-09-02 14:27:44,968 INFO [org.ovirt.engine.core.bll.storage.AddStoragePoolWithStoragesCommand] (ajp-/127.0.0.1:8009-6) [4e1e2715] Lock Acquired to object EngineLock [exclusiveLocks= key: ba2bec5d-7b0c-4fbc-941f-a2a62457907a value: POOL >, sharedLocks= ] >2012-09-02 14:27:44,983 INFO [org.ovirt.engine.core.bll.storage.AddStoragePoolWithStoragesCommand] (ajp-/127.0.0.1:8009-6) [4e1e2715] Running command: AddStoragePoolWithStoragesCommand internal: true. Entities affected : ID: ba2bec5d-7b0c-4fbc-941f-a2a62457907a Type: StoragePool >2012-09-02 14:27:45,116 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8009-6) [4e1e2715] START, ConnectStorageServerVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = ba2bec5d-7b0c-4fbc-941f-a2a62457907a, storageType = ISCSI, connectionList = [{ id: 86649454-f1a1-4ce2-83f6-cce30c31a7ad, connection: orion.qa.lab.tlv.redhat.com };]), log id: 707f588e >2012-09-02 14:27:45,208 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8009-6) [4e1e2715] FINISH, ConnectStorageServerVDSCommand, return: {86649454-f1a1-4ce2-83f6-cce30c31a7ad=0}, log id: 707f588e >2012-09-02 14:27:45,210 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStoragePoolVDSCommand] (ajp-/127.0.0.1:8009-6) [4e1e2715] START, CreateStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId=ba2bec5d-7b0c-4fbc-941f-a2a62457907a, storageType=ISCSI, storagePoolName=TestDataCenter, masterDomainId=5df9c9a3-d1d1-452c-9e01-4904708090dd, domainsIdList=[5df9c9a3-d1d1-452c-9e01-4904708090dd], masterVersion=1), log id: 37bcdd05 >2012-09-02 14:28:11,863 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStoragePoolVDSCommand] (ajp-/127.0.0.1:8009-6) [4e1e2715] FINISH, CreateStoragePoolVDSCommand, log id: 37bcdd05 >2012-09-02 14:28:11,888 INFO [org.ovirt.engine.core.bll.storage.AddStoragePoolWithStoragesCommand] (ajp-/127.0.0.1:8009-6) [4e1e2715] Lock freed to object EngineLock [exclusiveLocks= key: ba2bec5d-7b0c-4fbc-941f-a2a62457907a value: POOL >, sharedLocks= ] >2012-09-02 14:28:11,893 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp-/127.0.0.1:8009-6) [4b4f5c0d] Lock Acquired to object EngineLock [exclusiveLocks= key: 5df9c9a3-d1d1-452c-9e01-4904708090dd value: STORAGE >, sharedLocks= ] >2012-09-02 14:28:11,921 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp-/127.0.0.1:8009-6) [4b4f5c0d] Running command: ActivateStorageDomainCommand internal: true. Entities affected : ID: 5df9c9a3-d1d1-452c-9e01-4904708090dd Type: Storage >2012-09-02 14:28:11,950 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp-/127.0.0.1:8009-6) [4b4f5c0d] Lock freed to object EngineLock [exclusiveLocks= key: 5df9c9a3-d1d1-452c-9e01-4904708090dd value: STORAGE >, sharedLocks= ] >2012-09-02 14:28:11,950 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp-/127.0.0.1:8009-6) [4b4f5c0d] ActivateStorage Domain. Before Connect all hosts to pool. Time:9/2/12 2:28 PM >2012-09-02 14:28:11,985 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-4-thread-49) [4b4f5c0d] START, ConnectStorageServerVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = ba2bec5d-7b0c-4fbc-941f-a2a62457907a, storageType = ISCSI, connectionList = [{ id: 86649454-f1a1-4ce2-83f6-cce30c31a7ad, connection: orion.qa.lab.tlv.redhat.com };]), log id: 61b0519b >2012-09-02 14:28:12,075 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-4-thread-49) [4b4f5c0d] FINISH, ConnectStorageServerVDSCommand, return: {86649454-f1a1-4ce2-83f6-cce30c31a7ad=0}, log id: 61b0519b >2012-09-02 14:28:12,075 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp-/127.0.0.1:8009-6) [4b4f5c0d] ActivateStorage Domain. After Connect all hosts to pool. Time:9/2/12 2:28 PM >2012-09-02 14:28:12,079 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (ajp-/127.0.0.1:8009-6) [4b4f5c0d] START, ActivateStorageDomainVDSCommand(storagePoolId = ba2bec5d-7b0c-4fbc-941f-a2a62457907a, ignoreFailoverLimit = false, compatabilityVersion = null, storageDomainId = 5df9c9a3-d1d1-452c-9e01-4904708090dd), log id: 360b2944 >2012-09-02 14:28:12,095 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (ajp-/127.0.0.1:8009-6) [4af9d108] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: ba2bec5d-7b0c-4fbc-941f-a2a62457907a Type: StoragePool >2012-09-02 14:28:12,185 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (ajp-/127.0.0.1:8009-6) [4af9d108] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 14:28:12,264 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (ajp-/127.0.0.1:8009-6) [4af9d108] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = ba2bec5d-7b0c-4fbc-941f-a2a62457907a, vds_spm_id = 1, masterDomainId = 5df9c9a3-d1d1-452c-9e01-4904708090dd, masterVersion = 1), log id: 4d7cc558 >2012-09-02 14:28:13,663 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (ajp-/127.0.0.1:8009-6) [4af9d108] FINISH, ConnectStoragePoolVDSCommand, log id: 4d7cc558 >2012-09-02 14:28:13,684 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (ajp-/127.0.0.1:8009-6) [4af9d108] hostFromVds::Connected host to pool - selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Free, storage pool TestDataCenter >2012-09-02 14:28:13,745 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (ajp-/127.0.0.1:8009-6) [4af9d108] starting spm on vds green-vdsa.qa.lab.tlv.redhat.com, storage pool TestDataCenter, prevId -1, LVER -1 >2012-09-02 14:28:13,748 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (ajp-/127.0.0.1:8009-6) [4af9d108] START, SpmStartVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = ba2bec5d-7b0c-4fbc-941f-a2a62457907a, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6612faa0 >2012-09-02 14:28:13,775 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (ajp-/127.0.0.1:8009-6) [4af9d108] spmStart polling started: taskId = 9b046931-fd70-45fe-b241-76ce8ac251c7 >2012-09-02 14:28:18,077 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-48) [6daa7da2] START, SetVdsStatusVDSCommand(vdsId = 3880c572-f4f0-11e1-9a52-001a4a169705, status=NonResponsive, nonOperationalReason=NONE), log id: 5b653116 >2012-09-02 14:28:18,121 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-48) [6daa7da2] FINISH, SetVdsStatusVDSCommand, log id: 5b653116 >2012-09-02 14:28:20,109 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-74) [161e9a0] Running command: InitVdsOnUpCommand internal: true. >2012-09-02 14:28:20,121 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-74) [161e9a0] No string for UNASSIGNED type. Use default Log >2012-09-02 14:28:20,122 INFO [org.ovirt.engine.core.bll.MultipleActionsRunner] (pool-4-thread-48) [161e9a0] MultipleActionsRunner of type MigrateVmToServer invoked with no actions >2012-09-02 14:28:20,144 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-74) [2528e443] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 3880c572-f4f0-11e1-9a52-001a4a169705 Type: VDS >2012-09-02 14:28:20,226 INFO [org.ovirt.engine.core.bll.UpdateVdsGroupCommand] (QuartzScheduler_Worker-74) [53855601] Running command: UpdateVdsGroupCommand internal: true. Entities affected : ID: 99408929-82cf-4dc7-a532-9d998063fa95 Type: VdsGroups >2012-09-02 14:28:20,286 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-74) [2b7fbc48] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 3880c572-f4f0-11e1-9a52-001a4a169705 Type: VDS >2012-09-02 14:28:37,348 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (ajp-/127.0.0.1:8009-6) [4af9d108] spmStart polling ended: taskId = 9b046931-fd70-45fe-b241-76ce8ac251c7 task status = finished >2012-09-02 14:28:37,368 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (ajp-/127.0.0.1:8009-6) [4af9d108] spmStart polling ended. spm status: SPM >2012-09-02 14:28:37,370 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (ajp-/127.0.0.1:8009-6) [4af9d108] START, HSMClearTaskVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, taskId=9b046931-fd70-45fe-b241-76ce8ac251c7), log id: 45d9e221 >2012-09-02 14:28:37,389 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (ajp-/127.0.0.1:8009-6) [4af9d108] FINISH, HSMClearTaskVDSCommand, log id: 45d9e221 >2012-09-02 14:28:37,389 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (ajp-/127.0.0.1:8009-6) [4af9d108] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@53fd7f66, log id: 6612faa0 >2012-09-02 14:28:37,395 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (ajp-/127.0.0.1:8009-6) [4af9d108] Initialize Irs proxy from vds: 10.35.102.10 >2012-09-02 14:28:37,444 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (ajp-/127.0.0.1:8009-6) [4af9d108] FINISH, ActivateStorageDomainVDSCommand, log id: 360b2944 >2012-09-02 14:28:37,445 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp-/127.0.0.1:8009-6) [4af9d108] ActivateStorage Domain. After Activate storage domain in vds. Time:9/2/12 2:28 PM >2012-09-02 14:28:37,475 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-47) [4af9d108] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = ba2bec5d-7b0c-4fbc-941f-a2a62457907a, vds_spm_id = 1, masterDomainId = 5df9c9a3-d1d1-452c-9e01-4904708090dd, masterVersion = 1), log id: 3c304615 >2012-09-02 14:28:37,497 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-47) [4af9d108] FINISH, ConnectStoragePoolVDSCommand, log id: 3c304615 >2012-09-02 14:28:37,497 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp-/127.0.0.1:8009-6) [4af9d108] ActivateStorage Domain. After Refresh all pools . Time:9/2/12 2:28 PM >2012-09-02 14:28:37,549 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp-/127.0.0.1:8009-6) [4af9d108] ActivateStorage Domain. After change storage pool status in vds. Time:9/2/12 2:28 PM >2012-09-02 14:30:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-7) Autorecovering hosts is disabled, skipping >2012-09-02 14:30:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-7) Autorecovering storage domains is disabled, skipping >2012-09-02 14:35:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-6) Autorecovering hosts is disabled, skipping >2012-09-02 14:35:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-6) Autorecovering storage domains is disabled, skipping >2012-09-02 14:37:43,279 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8009-6) Checking if user admin@internal is an admin, result true >2012-09-02 14:37:43,279 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8009-6) Running command: LoginUserCommand internal: false. >2012-09-02 14:37:43,373 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8009-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2012-09-02 14:37:43,614 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8009-3) Checking if user admin@internal is an admin, result true >2012-09-02 14:37:43,614 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8009-3) Running command: LoginUserCommand internal: false. >2012-09-02 14:37:43,700 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8009-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2012-09-02 14:37:45,914 INFO [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (ajp-/127.0.0.1:8009-3) [3de6d839] Lock Acquired to object EngineLock [exclusiveLocks= key: testVm value: VM_NAME >, sharedLocks= ] >2012-09-02 14:37:45,978 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8009-3) [3de6d839] START, IsValidVDSCommand(storagePoolId = ba2bec5d-7b0c-4fbc-941f-a2a62457907a, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 29c0fad6 >2012-09-02 14:37:45,980 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8009-3) [3de6d839] FINISH, IsValidVDSCommand, return: true, log id: 29c0fad6 >2012-09-02 14:37:46,015 INFO [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (ajp-/127.0.0.1:8009-3) [3de6d839] Running command: AddVmFromScratchCommand internal: false. Entities affected : ID: b1600626-f4e9-11e1-b9cc-001a4a169705 Type: VdsGroups >2012-09-02 14:37:46,016 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8009-3) [3de6d839] START, IsValidVDSCommand(storagePoolId = ba2bec5d-7b0c-4fbc-941f-a2a62457907a, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3740ef >2012-09-02 14:37:46,017 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8009-3) [3de6d839] FINISH, IsValidVDSCommand, return: true, log id: 3740ef >2012-09-02 14:37:46,141 INFO [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (ajp-/127.0.0.1:8009-3) [3de6d839] Lock freed to object EngineLock [exclusiveLocks= key: testVm value: VM_NAME >, sharedLocks= ] >2012-09-02 14:37:48,090 INFO [org.ovirt.engine.core.bll.AddVmInterfaceCommand] (ajp-/127.0.0.1:8009-1) [4c60c048] Running command: AddVmInterfaceCommand internal: false. Entities affected : ID: a3e69725-8fb0-42bf-b04b-453700d9a958 Type: VM >2012-09-02 14:37:48,091 INFO [org.ovirt.engine.core.bll.MacPoolManager] (ajp-/127.0.0.1:8009-1) [4c60c048] MacPoolManager::allocateNewMac entered >2012-09-02 14:37:48,091 INFO [org.ovirt.engine.core.bll.MacPoolManager] (ajp-/127.0.0.1:8009-1) [4c60c048] MacPoolManager::allocateNewMac allocated mac = 00:1a:4a:23:61:69 >2012-09-02 14:37:49,323 INFO [org.ovirt.engine.core.bll.AddDiskCommand] (ajp-/127.0.0.1:8009-3) [69fcfe80] Lock Acquired to object EngineLock [exclusiveLocks= key: a3e69725-8fb0-42bf-b04b-453700d9a958 value: VM_DISK_BOOT >, sharedLocks= key: a3e69725-8fb0-42bf-b04b-453700d9a958 value: VM >] >2012-09-02 14:37:49,332 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8009-3) [69fcfe80] START, IsValidVDSCommand(storagePoolId = ba2bec5d-7b0c-4fbc-941f-a2a62457907a, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 609722a7 >2012-09-02 14:37:49,332 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8009-3) [69fcfe80] FINISH, IsValidVDSCommand, return: true, log id: 609722a7 >2012-09-02 14:37:49,336 WARN [org.ovirt.engine.core.bll.AddDiskCommand] (ajp-/127.0.0.1:8009-3) [69fcfe80] CanDoAction of action AddDisk failed. Reasons:VAR__ACTION__ADD,VAR__TYPE__VM_DISK,ACTION_TYPE_FAILED_DISK_SPACE_LOW >2012-09-02 14:37:49,336 INFO [org.ovirt.engine.core.bll.AddDiskCommand] (ajp-/127.0.0.1:8009-3) [69fcfe80] Lock freed to object EngineLock [exclusiveLocks= key: a3e69725-8fb0-42bf-b04b-453700d9a958 value: VM_DISK_BOOT >, sharedLocks= key: a3e69725-8fb0-42bf-b04b-453700d9a958 value: VM >] >2012-09-02 14:37:49,369 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (ajp-/127.0.0.1:8009-3) Operation Failed: [Cannot add Virtual Machine Disk. Low disk space on relevant Storage Domain.] >2012-09-02 14:38:42,831 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetDeviceListVDSCommand] (ajp-/127.0.0.1:8009-10) START, GetDeviceListVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storageType=ISCSI), log id: 44515a08 >2012-09-02 14:38:42,844 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetDeviceListVDSCommand] (ajp-/127.0.0.1:8009-3) START, GetDeviceListVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storageType=ISCSI), log id: 3ca568a4 >2012-09-02 14:38:43,866 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetDeviceListVDSCommand] (ajp-/127.0.0.1:8009-10) FINISH, GetDeviceListVDSCommand, return: [org.ovirt.engine.core.common.businessentities.LUNs@fd7af373, org.ovirt.engine.core.common.businessentities.LUNs@602c4909], log id: 44515a08 >2012-09-02 14:38:44,431 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetDeviceListVDSCommand] (ajp-/127.0.0.1:8009-3) FINISH, GetDeviceListVDSCommand, return: [org.ovirt.engine.core.common.businessentities.LUNs@fd7af373, org.ovirt.engine.core.common.businessentities.LUNs@602c4909], log id: 3ca568a4 >2012-09-02 14:40:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-78) Autorecovering hosts is disabled, skipping >2012-09-02 14:40:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-78) Autorecovering storage domains is disabled, skipping >2012-09-02 14:45:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-98) Autorecovering hosts is disabled, skipping >2012-09-02 14:45:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-98) Autorecovering storage domains is disabled, skipping >2012-09-02 14:50:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-95) Autorecovering hosts is disabled, skipping >2012-09-02 14:50:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-95) Autorecovering storage domains is disabled, skipping >2012-09-02 14:55:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-88) Autorecovering hosts is disabled, skipping >2012-09-02 14:55:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-88) Autorecovering storage domains is disabled, skipping >2012-09-02 14:58:15,922 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) domain 5df9c9a3-d1d1-452c-9e01-4904708090dd in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 14:58:28,830 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-59) Command SpmStatusVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 14:58:28,834 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-59) [61d5001e] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: ba2bec5d-7b0c-4fbc-941f-a2a62457907a Type: StoragePool >2012-09-02 14:58:29,159 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-59) [61d5001e] Command SpmStatusVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 14:58:29,159 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [61d5001e] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 14:58:29,159 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [61d5001e] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 14:58:29,159 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [61d5001e] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 14:58:29,265 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-59) [1138eff2] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 5df9c9a3-d1d1-452c-9e01-4904708090dd Type: Storage >2012-09-02 14:58:29,306 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-59) [1138eff2] START, ResetIrsVDSCommand(storagePoolId = ba2bec5d-7b0c-4fbc-941f-a2a62457907a, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = 10.35.102.10, vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, ignoreStopFailed = true), log id: 2bcdbdba >2012-09-02 14:58:29,308 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-59) [1138eff2] START, SpmStopVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = ba2bec5d-7b0c-4fbc-941f-a2a62457907a), log id: 1886d589 >2012-09-02 14:58:29,328 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-59) [1138eff2] SpmStopVDSCommand::Stopping SPM on vds green-vdsa.qa.lab.tlv.redhat.com, pool id ba2bec5d-7b0c-4fbc-941f-a2a62457907a >2012-09-02 14:58:29,848 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-59) [1138eff2] could not stop spm of pool ba2bec5d-7b0c-4fbc-941f-a2a62457907a on vds b32eacf0-f4e9-11e1-973c-001a4a169705 - reason: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2012-09-02 14:58:29,848 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-59) [1138eff2] FINISH, SpmStopVDSCommand, log id: 1886d589 >2012-09-02 14:58:29,972 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-59) [1138eff2] FINISH, ResetIrsVDSCommand, log id: 2bcdbdba >2012-09-02 14:58:29,984 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-59) [1138eff2] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = ba2bec5d-7b0c-4fbc-941f-a2a62457907a, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 63eaae2e >2012-09-02 14:58:29,985 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [1138eff2] clearing cache for problematic entities in pool ba2bec5d-7b0c-4fbc-941f-a2a62457907a >2012-09-02 14:58:29,985 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-59) [1138eff2] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 63eaae2e >2012-09-02 14:58:30,389 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-63) ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 14:58:32,996 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-67) [22c21747] Running command: InitVdsOnUpCommand internal: true. >2012-09-02 14:58:33,075 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-67) [194ad921] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: ba2bec5d-7b0c-4fbc-941f-a2a62457907a Type: StoragePool >2012-09-02 14:58:33,076 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (QuartzScheduler_Worker-67) [194ad921] START, ConnectStorageServerVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = ba2bec5d-7b0c-4fbc-941f-a2a62457907a, storageType = ISCSI, connectionList = [{ id: 86649454-f1a1-4ce2-83f6-cce30c31a7ad, connection: orion.qa.lab.tlv.redhat.com };]), log id: 3729768 >2012-09-02 14:58:33,157 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (QuartzScheduler_Worker-67) [194ad921] FINISH, ConnectStorageServerVDSCommand, return: {86649454-f1a1-4ce2-83f6-cce30c31a7ad=0}, log id: 3729768 >2012-09-02 14:58:33,157 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-67) [194ad921] Host green-vdsa.qa.lab.tlv.redhat.com storage connection was succeeded >2012-09-02 14:58:33,181 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-67) [194ad921] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = ba2bec5d-7b0c-4fbc-941f-a2a62457907a, vds_spm_id = 1, masterDomainId = 5df9c9a3-d1d1-452c-9e01-4904708090dd, masterVersion = 1), log id: 13921c1f >2012-09-02 14:58:34,359 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-67) [194ad921] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=ba2bec5d-7b0c-4fbc-941f-a2a62457907a, msdUUID=5df9c9a3-d1d1-452c-9e01-4904708090dd' > > >2012-09-02 14:58:34,359 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-67) [194ad921] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 14:58:34,359 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-67) [194ad921] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=ba2bec5d-7b0c-4fbc-941f-a2a62457907a, msdUUID=5df9c9a3-d1d1-452c-9e01-4904708090dd' >2012-09-02 14:58:34,359 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-67) [194ad921] FINISH, ConnectStoragePoolVDSCommand, log id: 13921c1f >2012-09-02 14:58:34,359 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-67) [194ad921] Could not connect host green-vdsa.qa.lab.tlv.redhat.com to pool TestDataCenter >2012-09-02 14:58:34,415 INFO [org.ovirt.engine.core.bll.MultipleActionsRunner] (pool-4-thread-47) [194ad921] MultipleActionsRunner of type MigrateVmToServer invoked with no actions >2012-09-02 14:58:34,433 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-67) [3b44010f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: b32eacf0-f4e9-11e1-973c-001a4a169705 Type: VDS >2012-09-02 14:58:34,450 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-67) [fb71dbe] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: b32eacf0-f4e9-11e1-973c-001a4a169705 Type: VDS >2012-09-02 14:58:38,975 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-70) domain 5df9c9a3-d1d1-452c-9e01-4904708090dd in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 14:58:40,427 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [2b7fbc48] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 14:58:40,619 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-74) [2b7fbc48] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = ba2bec5d-7b0c-4fbc-941f-a2a62457907a, vds_spm_id = 1, masterDomainId = 5df9c9a3-d1d1-452c-9e01-4904708090dd, masterVersion = 1), log id: 666867d5 >2012-09-02 14:58:41,440 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-74) [2b7fbc48] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=ba2bec5d-7b0c-4fbc-941f-a2a62457907a, msdUUID=5df9c9a3-d1d1-452c-9e01-4904708090dd' > > >2012-09-02 14:58:41,440 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-74) [2b7fbc48] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 14:58:41,440 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-74) [2b7fbc48] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=ba2bec5d-7b0c-4fbc-941f-a2a62457907a, msdUUID=5df9c9a3-d1d1-452c-9e01-4904708090dd' >2012-09-02 14:58:41,440 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-74) [2b7fbc48] FINISH, ConnectStoragePoolVDSCommand, log id: 666867d5 >2012-09-02 14:58:41,440 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [2b7fbc48] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 14:58:41,440 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [2b7fbc48] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=ba2bec5d-7b0c-4fbc-941f-a2a62457907a, msdUUID=5df9c9a3-d1d1-452c-9e01-4904708090dd' >2012-09-02 14:58:41,588 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-74) [59641388] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 5df9c9a3-d1d1-452c-9e01-4904708090dd Type: Storage >2012-09-02 14:58:41,639 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-74) [59641388] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = ba2bec5d-7b0c-4fbc-941f-a2a62457907a, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3983a11b >2012-09-02 14:58:41,639 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [59641388] clearing cache for problematic entities in pool ba2bec5d-7b0c-4fbc-941f-a2a62457907a >2012-09-02 14:58:41,639 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-74) [59641388] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3983a11b >2012-09-02 14:58:49,161 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) domain 5df9c9a3-d1d1-452c-9e01-4904708090dd in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 14:58:51,710 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 14:58:51,732 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-98) START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = ba2bec5d-7b0c-4fbc-941f-a2a62457907a, vds_spm_id = 1, masterDomainId = 5df9c9a3-d1d1-452c-9e01-4904708090dd, masterVersion = 1), log id: 3d6cd571 >2012-09-02 14:58:52,550 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-98) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=ba2bec5d-7b0c-4fbc-941f-a2a62457907a, msdUUID=5df9c9a3-d1d1-452c-9e01-4904708090dd' > > >2012-09-02 14:58:52,550 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-98) Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 14:58:52,550 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-98) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=ba2bec5d-7b0c-4fbc-941f-a2a62457907a, msdUUID=5df9c9a3-d1d1-452c-9e01-4904708090dd' >2012-09-02 14:58:52,550 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-98) FINISH, ConnectStoragePoolVDSCommand, log id: 3d6cd571 >2012-09-02 14:58:52,550 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 14:58:52,550 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=ba2bec5d-7b0c-4fbc-941f-a2a62457907a, msdUUID=5df9c9a3-d1d1-452c-9e01-4904708090dd' >2012-09-02 14:58:52,669 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-98) [5bec48ce] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 5df9c9a3-d1d1-452c-9e01-4904708090dd Type: Storage >2012-09-02 14:58:52,710 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-98) [5bec48ce] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = ba2bec5d-7b0c-4fbc-941f-a2a62457907a, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4ce217d3 >2012-09-02 14:58:52,712 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [5bec48ce] clearing cache for problematic entities in pool ba2bec5d-7b0c-4fbc-941f-a2a62457907a >2012-09-02 14:58:52,712 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-98) [5bec48ce] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4ce217d3 >2012-09-02 14:58:59,340 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-3) domain 5df9c9a3-d1d1-452c-9e01-4904708090dd in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 14:59:02,760 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 14:59:03,113 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-5) START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = ba2bec5d-7b0c-4fbc-941f-a2a62457907a, vds_spm_id = 1, masterDomainId = 5df9c9a3-d1d1-452c-9e01-4904708090dd, masterVersion = 1), log id: 44ecafcd >2012-09-02 14:59:03,932 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-5) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=ba2bec5d-7b0c-4fbc-941f-a2a62457907a, msdUUID=5df9c9a3-d1d1-452c-9e01-4904708090dd' > > >2012-09-02 14:59:03,932 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-5) Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 14:59:03,932 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-5) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=ba2bec5d-7b0c-4fbc-941f-a2a62457907a, msdUUID=5df9c9a3-d1d1-452c-9e01-4904708090dd' >2012-09-02 14:59:03,932 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-5) FINISH, ConnectStoragePoolVDSCommand, log id: 44ecafcd >2012-09-02 14:59:03,932 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 14:59:03,932 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=ba2bec5d-7b0c-4fbc-941f-a2a62457907a, msdUUID=5df9c9a3-d1d1-452c-9e01-4904708090dd' >2012-09-02 14:59:04,132 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-5) [63b152d2] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 5df9c9a3-d1d1-452c-9e01-4904708090dd Type: Storage >2012-09-02 14:59:04,167 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-5) [63b152d2] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = ba2bec5d-7b0c-4fbc-941f-a2a62457907a, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6db119e0 >2012-09-02 14:59:04,168 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [63b152d2] clearing cache for problematic entities in pool ba2bec5d-7b0c-4fbc-941f-a2a62457907a >2012-09-02 14:59:04,168 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-5) [63b152d2] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6db119e0 >2012-09-02 14:59:08,608 INFO [org.ovirt.engine.core.bll.storage.RemoveStoragePoolCommand] (ajp-/127.0.0.1:8009-4) [3b9c71d4] Running command: RemoveStoragePoolCommand internal: false. Entities affected : ID: ba2bec5d-7b0c-4fbc-941f-a2a62457907a Type: StoragePool >2012-09-02 15:00:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-83) [28a39dff] Autorecovering hosts is disabled, skipping >2012-09-02 15:00:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-83) [28a39dff] Autorecovering storage domains is disabled, skipping >2012-09-02 15:03:41,964 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-77) DbUserCacheManager::refreshAllUserData() - entered >2012-09-02 15:05:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-92) Autorecovering hosts is disabled, skipping >2012-09-02 15:05:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-92) Autorecovering storage domains is disabled, skipping >2012-09-02 15:10:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-87) Autorecovering hosts is disabled, skipping >2012-09-02 15:10:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-87) Autorecovering storage domains is disabled, skipping >2012-09-02 15:10:38,867 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8009-8) Checking if user admin@internal is an admin, result true >2012-09-02 15:10:38,867 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8009-8) Running command: LoginUserCommand internal: false. >2012-09-02 15:10:38,921 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8009-8) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2012-09-02 15:10:39,150 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8009-7) Checking if user admin@internal is an admin, result true >2012-09-02 15:10:39,150 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8009-7) Running command: LoginUserCommand internal: false. >2012-09-02 15:10:39,189 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8009-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2012-09-02 15:10:40,958 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (ajp-/127.0.0.1:8009-10) [6c20432f] START, ValidateStorageServerConnectionVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = ISCSI, connectionList = [{ id: null, connection: 10.35.64.10 };]), log id: 4426cd0a >2012-09-02 15:10:40,992 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (ajp-/127.0.0.1:8009-10) [6c20432f] FINISH, ValidateStorageServerConnectionVDSCommand, return: {00000000-0000-0000-0000-000000000000=0}, log id: 4426cd0a >2012-09-02 15:10:41,012 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (ajp-/127.0.0.1:8009-10) [6c20432f] Running command: ConnectStorageToVdsCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2012-09-02 15:10:41,013 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8009-10) [6c20432f] START, ConnectStorageServerVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = ISCSI, connectionList = [{ id: null, connection: 10.35.64.10 };]), log id: 7d4d502e >2012-09-02 15:10:41,720 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8009-10) [6c20432f] FINISH, ConnectStorageServerVDSCommand, return: {00000000-0000-0000-0000-000000000000=0}, log id: 7d4d502e >2012-09-02 15:10:41,749 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetDeviceListVDSCommand] (ajp-/127.0.0.1:8009-10) START, GetDeviceListVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storageType=ISCSI), log id: 3c754e77 >2012-09-02 15:10:42,836 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetDeviceListVDSCommand] (ajp-/127.0.0.1:8009-10) FINISH, GetDeviceListVDSCommand, return: [org.ovirt.engine.core.common.businessentities.LUNs@81090923, org.ovirt.engine.core.common.businessentities.LUNs@ff187c65, org.ovirt.engine.core.common.businessentities.LUNs@2f995e1, org.ovirt.engine.core.common.businessentities.LUNs@42014f82, org.ovirt.engine.core.common.businessentities.LUNs@c010c2c4], log id: 3c754e77 >2012-09-02 15:10:42,914 WARN [org.ovirt.engine.core.bll.storage.AddSANStorageDomainCommand] (ajp-/127.0.0.1:8009-10) [65b42126] CanDoAction of action AddSANStorageDomain failed. Reasons:VAR__ACTION__ADD,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_STORAGE_DOMAIN_FORMAT_ILLEGAL_HOST,$storageFormat V3 >2012-09-02 15:10:42,926 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (ajp-/127.0.0.1:8009-10) Operation Failed: [Cannot add Storage. Storage format V3 is not supported on the selected host version.] >2012-09-02 15:11:08,067 INFO [org.ovirt.engine.core.bll.storage.AddEmptyStoragePoolCommand] (ajp-/127.0.0.1:8009-8) [41abfe27] Running command: AddEmptyStoragePoolCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2012-09-02 15:11:16,320 INFO [org.ovirt.engine.core.bll.UpdateVdsGroupCommand] (ajp-/127.0.0.1:8009-1) [41c2adf0] Running command: UpdateVdsGroupCommand internal: false. Entities affected : ID: b1600626-f4e9-11e1-b9cc-001a4a169705 Type: VdsGroups >2012-09-02 15:11:16,326 INFO [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp-/127.0.0.1:8009-1) [33d45b50] Running command: AddVdsSpmIdCommand internal: true. Entities affected : ID: b32eacf0-f4e9-11e1-973c-001a4a169705 Type: VDS >2012-09-02 15:11:16,328 INFO [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp-/127.0.0.1:8009-1) [33d45b50] Lock freed to object EngineLock [exclusiveLocks= key: bd560b80-c245-46b3-ad8c-b142a3460cf6 value: REGISTER_VDS >, sharedLocks= ] >2012-09-02 15:11:21,756 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8009-1) Checking if user admin@internal is an admin, result true >2012-09-02 15:11:21,756 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8009-1) Running command: LoginUserCommand internal: false. >2012-09-02 15:11:21,799 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8009-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2012-09-02 15:11:21,993 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8009-11) Checking if user admin@internal is an admin, result true >2012-09-02 15:11:21,993 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8009-11) Running command: LoginUserCommand internal: false. >2012-09-02 15:11:22,067 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8009-11) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2012-09-02 15:11:23,928 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (ajp-/127.0.0.1:8009-6) [3f8e5ba0] START, ValidateStorageServerConnectionVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = ISCSI, connectionList = [{ id: null, connection: 10.35.64.10 };]), log id: 52e874a0 >2012-09-02 15:11:23,962 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (ajp-/127.0.0.1:8009-6) [3f8e5ba0] FINISH, ValidateStorageServerConnectionVDSCommand, return: {00000000-0000-0000-0000-000000000000=0}, log id: 52e874a0 >2012-09-02 15:11:23,995 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (ajp-/127.0.0.1:8009-6) [3f8e5ba0] Running command: ConnectStorageToVdsCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2012-09-02 15:11:23,996 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8009-6) [3f8e5ba0] START, ConnectStorageServerVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = ISCSI, connectionList = [{ id: null, connection: 10.35.64.10 };]), log id: 3284e8d7 >2012-09-02 15:11:24,110 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8009-6) [3f8e5ba0] FINISH, ConnectStorageServerVDSCommand, return: {00000000-0000-0000-0000-000000000000=0}, log id: 3284e8d7 >2012-09-02 15:11:24,142 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetDeviceListVDSCommand] (ajp-/127.0.0.1:8009-6) START, GetDeviceListVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storageType=ISCSI), log id: 1b2e8975 >2012-09-02 15:11:25,183 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetDeviceListVDSCommand] (ajp-/127.0.0.1:8009-6) FINISH, GetDeviceListVDSCommand, return: [org.ovirt.engine.core.common.businessentities.LUNs@81090923, org.ovirt.engine.core.common.businessentities.LUNs@ff187c65, org.ovirt.engine.core.common.businessentities.LUNs@2f995e1, org.ovirt.engine.core.common.businessentities.LUNs@42014f82, org.ovirt.engine.core.common.businessentities.LUNs@c010c2c4], log id: 1b2e8975 >2012-09-02 15:11:25,244 INFO [org.ovirt.engine.core.bll.storage.AddSANStorageDomainCommand] (ajp-/127.0.0.1:8009-6) [1dd0a99] Running command: AddSANStorageDomainCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2012-09-02 15:11:25,267 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVGVDSCommand] (ajp-/127.0.0.1:8009-6) [1dd0a99] START, CreateVGVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storageDomainId=c2eb314a-4d7a-4171-981f-bb70d1d4d4c6, deviceList=[gadi-lun1], force=false), log id: 71d769d4 >2012-09-02 15:11:25,302 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (ajp-/127.0.0.1:8009-6) [1dd0a99] Failed in CreateVGVDS method >2012-09-02 15:11:25,302 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (ajp-/127.0.0.1:8009-6) [1dd0a99] Error code InvalidPhysDev and error message VDSGenericException: VDSErrorException: Failed to CreateVGVDS, error = Invalid physical device: ('gadi-lun1',) >2012-09-02 15:11:25,302 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (ajp-/127.0.0.1:8009-6) [1dd0a99] Command org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVGVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.irsbroker.OneUuidReturnForXmlRpc >mUuid Null >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 606 >mMessage Invalid physical device: ('gadi-lun1',) > > >2012-09-02 15:11:25,302 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (ajp-/127.0.0.1:8009-6) [1dd0a99] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:11:25,303 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (ajp-/127.0.0.1:8009-6) [1dd0a99] Command CreateVGVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to CreateVGVDS, error = Invalid physical device: ('gadi-lun1',) >2012-09-02 15:11:25,303 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVGVDSCommand] (ajp-/127.0.0.1:8009-6) [1dd0a99] FINISH, CreateVGVDSCommand, log id: 71d769d4 >2012-09-02 15:11:25,303 ERROR [org.ovirt.engine.core.bll.storage.AddSANStorageDomainCommand] (ajp-/127.0.0.1:8009-6) [1dd0a99] Command org.ovirt.engine.core.bll.storage.AddSANStorageDomainCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to CreateVGVDS, error = Invalid physical device: ('gadi-lun1',) >2012-09-02 15:11:25,305 INFO [org.ovirt.engine.core.bll.storage.AddSANStorageDomainCommand] (ajp-/127.0.0.1:8009-6) [1dd0a99] Command [id=0de05eca-b935-454b-a0b0-bd0dec0aefa7]: Compensating NEW_ENTITY_ID of org.ovirt.engine.core.common.businessentities.storage_domain_dynamic; snapshot: c2eb314a-4d7a-4171-981f-bb70d1d4d4c6. >2012-09-02 15:11:25,307 INFO [org.ovirt.engine.core.bll.storage.AddSANStorageDomainCommand] (ajp-/127.0.0.1:8009-6) [1dd0a99] Command [id=0de05eca-b935-454b-a0b0-bd0dec0aefa7]: Compensating NEW_ENTITY_ID of org.ovirt.engine.core.common.businessentities.storage_domain_static; snapshot: c2eb314a-4d7a-4171-981f-bb70d1d4d4c6. >2012-09-02 15:11:25,321 ERROR [org.ovirt.engine.core.bll.storage.AddSANStorageDomainCommand] (ajp-/127.0.0.1:8009-6) [1dd0a99] Transaction rolled-back for command: org.ovirt.engine.core.bll.storage.AddSANStorageDomainCommand. >2012-09-02 15:11:25,336 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (ajp-/127.0.0.1:8009-6) Operation Failed: [] >2012-09-02 15:12:54,255 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8009-7) Checking if user admin@internal is an admin, result true >2012-09-02 15:12:54,256 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8009-7) Running command: LoginUserCommand internal: false. >2012-09-02 15:12:54,293 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8009-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2012-09-02 15:12:54,496 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8009-6) Checking if user admin@internal is an admin, result true >2012-09-02 15:12:54,497 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8009-6) Running command: LoginUserCommand internal: false. >2012-09-02 15:12:54,535 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8009-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2012-09-02 15:12:56,299 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (ajp-/127.0.0.1:8009-9) [3b438002] START, ValidateStorageServerConnectionVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = ISCSI, connectionList = [{ id: null, connection: 10.35.64.10 };]), log id: 6167d703 >2012-09-02 15:12:56,320 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (ajp-/127.0.0.1:8009-9) [3b438002] FINISH, ValidateStorageServerConnectionVDSCommand, return: {00000000-0000-0000-0000-000000000000=0}, log id: 6167d703 >2012-09-02 15:12:56,364 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (ajp-/127.0.0.1:8009-9) [3b438002] Running command: ConnectStorageToVdsCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2012-09-02 15:12:56,367 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8009-9) [3b438002] START, ConnectStorageServerVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = ISCSI, connectionList = [{ id: null, connection: 10.35.64.10 };]), log id: 6a407ed6 >2012-09-02 15:12:56,462 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8009-9) [3b438002] FINISH, ConnectStorageServerVDSCommand, return: {00000000-0000-0000-0000-000000000000=0}, log id: 6a407ed6 >2012-09-02 15:12:56,475 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetDeviceListVDSCommand] (ajp-/127.0.0.1:8009-9) START, GetDeviceListVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storageType=ISCSI), log id: 74fe247a >2012-09-02 15:12:57,452 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetDeviceListVDSCommand] (ajp-/127.0.0.1:8009-9) FINISH, GetDeviceListVDSCommand, return: [org.ovirt.engine.core.common.businessentities.LUNs@81090923, org.ovirt.engine.core.common.businessentities.LUNs@ff187c65, org.ovirt.engine.core.common.businessentities.LUNs@2f995e1, org.ovirt.engine.core.common.businessentities.LUNs@42014f82, org.ovirt.engine.core.common.businessentities.LUNs@c010c2c4], log id: 74fe247a >2012-09-02 15:12:57,543 INFO [org.ovirt.engine.core.bll.storage.AddSANStorageDomainCommand] (ajp-/127.0.0.1:8009-9) [3f525709] Running command: AddSANStorageDomainCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2012-09-02 15:12:57,566 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVGVDSCommand] (ajp-/127.0.0.1:8009-9) [3f525709] START, CreateVGVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storageDomainId=f1bfec25-07f6-4c93-b46b-7caecf9d25f1, deviceList=[1gadi-lun213465860], force=false), log id: 7959eb50 >2012-09-02 15:12:58,506 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVGVDSCommand] (ajp-/127.0.0.1:8009-9) [3f525709] FINISH, CreateVGVDSCommand, return: ro21pW-0IKJ-Ovet-lsVk-EU1T-L8Ls-Ifprsf, log id: 7959eb50 >2012-09-02 15:12:58,507 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (ajp-/127.0.0.1:8009-9) [3f525709] START, CreateStorageDomainVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storageDomain=org.ovirt.engine.core.common.businessentities.storage_domain_static@8f5830b5, args=ro21pW-0IKJ-Ovet-lsVk-EU1T-L8Ls-Ifprsf), log id: 61eee5fd >2012-09-02 15:13:04,783 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (ajp-/127.0.0.1:8009-9) [3f525709] FINISH, CreateStorageDomainVDSCommand, log id: 61eee5fd >2012-09-02 15:13:04,788 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStorageDomainStatsVDSCommand] (ajp-/127.0.0.1:8009-9) [3f525709] START, GetStorageDomainStatsVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storageDomainId=f1bfec25-07f6-4c93-b46b-7caecf9d25f1), log id: 1e4a1098 >2012-09-02 15:13:05,002 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStorageDomainStatsVDSCommand] (ajp-/127.0.0.1:8009-9) [3f525709] FINISH, GetStorageDomainStatsVDSCommand, return: org.ovirt.engine.core.common.businessentities.storage_domains@f7a548f9, log id: 1e4a1098 >2012-09-02 15:13:05,055 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVGInfoVDSCommand] (ajp-/127.0.0.1:8009-9) [3f525709] START, GetVGInfoVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, VGID=ro21pW-0IKJ-Ovet-lsVk-EU1T-L8Ls-Ifprsf), log id: 2a39090f >2012-09-02 15:13:05,093 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVGInfoVDSCommand] (ajp-/127.0.0.1:8009-9) [3f525709] FINISH, GetVGInfoVDSCommand, return: [org.ovirt.engine.core.common.businessentities.LUNs@d51e4b82], log id: 2a39090f >2012-09-02 15:13:05,510 WARN [org.ovirt.engine.core.bll.storage.AttachStorageDomainToPoolCommand] (ajp-/127.0.0.1:8009-7) [3e747d5d] CanDoAction of action AttachStorageDomainToPool failed. Reasons:VAR__TYPE__STORAGE__DOMAIN,VAR__ACTION__ATTACH,ERROR_CANNOT_ATTACH_STORAGE_DOMAIN_STORAGE_TYPE_NOT_MATCH >2012-09-02 15:13:05,523 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (ajp-/127.0.0.1:8009-7) Operation Failed: [Cannot attach storage to Data Center. Storage type doesn't match the Data Center type.] >2012-09-02 15:13:25,754 INFO [org.ovirt.engine.core.bll.storage.UpdateStoragePoolCommand] (ajp-/127.0.0.1:8009-10) [140b891c] Running command: UpdateStoragePoolCommand internal: false. Entities affected : ID: bd560b80-c245-46b3-ad8c-b142a3460cf6 Type: StoragePool >2012-09-02 15:14:15,171 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8009-7) Checking if user admin@internal is an admin, result true >2012-09-02 15:14:15,172 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8009-7) Running command: LoginUserCommand internal: false. >2012-09-02 15:14:15,306 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8009-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2012-09-02 15:14:15,499 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8009-6) Checking if user admin@internal is an admin, result true >2012-09-02 15:14:15,499 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8009-6) Running command: LoginUserCommand internal: false. >2012-09-02 15:14:15,534 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8009-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2012-09-02 15:14:17,538 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (ajp-/127.0.0.1:8009-2) [19a4ddc9] START, ValidateStorageServerConnectionVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = ISCSI, connectionList = [{ id: null, connection: 10.35.64.10 };]), log id: 7ed03cb4 >2012-09-02 15:14:17,559 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (ajp-/127.0.0.1:8009-2) [19a4ddc9] FINISH, ValidateStorageServerConnectionVDSCommand, return: {00000000-0000-0000-0000-000000000000=0}, log id: 7ed03cb4 >2012-09-02 15:14:17,592 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (ajp-/127.0.0.1:8009-2) [19a4ddc9] Running command: ConnectStorageToVdsCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2012-09-02 15:14:17,594 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8009-2) [19a4ddc9] START, ConnectStorageServerVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = ISCSI, connectionList = [{ id: null, connection: 10.35.64.10 };]), log id: fdc72ce >2012-09-02 15:14:17,664 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8009-2) [19a4ddc9] FINISH, ConnectStorageServerVDSCommand, return: {00000000-0000-0000-0000-000000000000=0}, log id: fdc72ce >2012-09-02 15:14:17,692 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetDeviceListVDSCommand] (ajp-/127.0.0.1:8009-2) START, GetDeviceListVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storageType=ISCSI), log id: 3e066497 >2012-09-02 15:14:18,661 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetDeviceListVDSCommand] (ajp-/127.0.0.1:8009-2) FINISH, GetDeviceListVDSCommand, return: [org.ovirt.engine.core.common.businessentities.LUNs@81090923, org.ovirt.engine.core.common.businessentities.LUNs@ff187c65, org.ovirt.engine.core.common.businessentities.LUNs@2f995e1, org.ovirt.engine.core.common.businessentities.LUNs@42014f82, org.ovirt.engine.core.common.businessentities.LUNs@b39e7c16], log id: 3e066497 >2012-09-02 15:14:18,759 WARN [org.ovirt.engine.core.bll.storage.AddSANStorageDomainCommand] (ajp-/127.0.0.1:8009-2) [78aed64] CanDoAction of action AddSANStorageDomain failed. Reasons:VAR__ACTION__ADD,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_STORAGE_DOMAIN_NAME_ALREADY_EXIST >2012-09-02 15:14:18,776 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (ajp-/127.0.0.1:8009-2) Operation Failed: [Cannot add Storage. The Storage Domain name is already in use.] >2012-09-02 15:14:33,315 INFO [org.ovirt.engine.core.bll.storage.RemoveStorageDomainCommand] (ajp-/127.0.0.1:8009-6) [130698f5] Lock Acquired to object EngineLock [exclusiveLocks= key: f1bfec25-07f6-4c93-b46b-7caecf9d25f1 value: STORAGE >, sharedLocks= ] >2012-09-02 15:14:33,343 INFO [org.ovirt.engine.core.bll.storage.RemoveStorageDomainCommand] (ajp-/127.0.0.1:8009-6) [130698f5] Running command: RemoveStorageDomainCommand internal: false. Entities affected : ID: f1bfec25-07f6-4c93-b46b-7caecf9d25f1 Type: Storage >2012-09-02 15:14:33,345 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8009-6) [130698f5] START, ConnectStorageServerVDSCommand(vdsId = 3880c572-f4f0-11e1-9a52-001a4a169705, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = ISCSI, connectionList = [{ id: 5b2afcc4-20c4-44ff-b786-5b7029030f63, connection: 10.35.64.10 };]), log id: 32bc461d >2012-09-02 15:14:33,985 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8009-6) [130698f5] FINISH, ConnectStorageServerVDSCommand, return: {5b2afcc4-20c4-44ff-b786-5b7029030f63=0}, log id: 32bc461d >2012-09-02 15:14:33,987 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FormatStorageDomainVDSCommand] (ajp-/127.0.0.1:8009-6) [130698f5] START, FormatStorageDomainVDSCommand(vdsId = 3880c572-f4f0-11e1-9a52-001a4a169705, storageDomainId=f1bfec25-07f6-4c93-b46b-7caecf9d25f1), log id: 3c49f62 >2012-09-02 15:14:38,728 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FormatStorageDomainVDSCommand] (ajp-/127.0.0.1:8009-6) [130698f5] FINISH, FormatStorageDomainVDSCommand, log id: 3c49f62 >2012-09-02 15:14:38,729 WARN [org.ovirt.engine.core.dal.dbbroker.generic.DBConfigUtils] (ajp-/127.0.0.1:8009-6) [130698f5] Didnt find the value of IsNeedSupportForOldVgAPI in DB for version 3.1 - using default: false >2012-09-02 15:14:38,734 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (ajp-/127.0.0.1:8009-6) [130698f5] START, DisconnectStorageServerVDSCommand(vdsId = 3880c572-f4f0-11e1-9a52-001a4a169705, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = ISCSI, connectionList = [{ id: 5b2afcc4-20c4-44ff-b786-5b7029030f63, connection: 10.35.64.10 };]), log id: 49178c65 >2012-09-02 15:14:38,863 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (ajp-/127.0.0.1:8009-6) [130698f5] FINISH, DisconnectStorageServerVDSCommand, return: {5b2afcc4-20c4-44ff-b786-5b7029030f63=0}, log id: 49178c65 >2012-09-02 15:14:38,883 INFO [org.ovirt.engine.core.bll.storage.RemoveStorageDomainCommand] (ajp-/127.0.0.1:8009-6) [130698f5] Lock freed to object EngineLock [exclusiveLocks= key: f1bfec25-07f6-4c93-b46b-7caecf9d25f1 value: STORAGE >, sharedLocks= ] >2012-09-02 15:14:44,244 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8009-6) Checking if user admin@internal is an admin, result true >2012-09-02 15:14:44,244 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8009-6) Running command: LoginUserCommand internal: false. >2012-09-02 15:14:44,278 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8009-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2012-09-02 15:14:45,265 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8009-11) Checking if user admin@internal is an admin, result true >2012-09-02 15:14:45,265 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8009-11) Running command: LoginUserCommand internal: false. >2012-09-02 15:14:45,336 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8009-11) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2012-09-02 15:14:47,082 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (ajp-/127.0.0.1:8009-2) [372cee4d] START, ValidateStorageServerConnectionVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = ISCSI, connectionList = [{ id: null, connection: 10.35.64.10 };]), log id: 35d001e2 >2012-09-02 15:14:47,101 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (ajp-/127.0.0.1:8009-2) [372cee4d] FINISH, ValidateStorageServerConnectionVDSCommand, return: {00000000-0000-0000-0000-000000000000=0}, log id: 35d001e2 >2012-09-02 15:14:47,193 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (ajp-/127.0.0.1:8009-2) [372cee4d] Running command: ConnectStorageToVdsCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2012-09-02 15:14:47,194 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8009-2) [372cee4d] START, ConnectStorageServerVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = ISCSI, connectionList = [{ id: null, connection: 10.35.64.10 };]), log id: 17bee1dc >2012-09-02 15:14:47,264 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8009-2) [372cee4d] FINISH, ConnectStorageServerVDSCommand, return: {00000000-0000-0000-0000-000000000000=0}, log id: 17bee1dc >2012-09-02 15:14:47,292 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetDeviceListVDSCommand] (ajp-/127.0.0.1:8009-2) START, GetDeviceListVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storageType=ISCSI), log id: 5ef4ed81 >2012-09-02 15:14:48,316 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetDeviceListVDSCommand] (ajp-/127.0.0.1:8009-2) FINISH, GetDeviceListVDSCommand, return: [org.ovirt.engine.core.common.businessentities.LUNs@81090923, org.ovirt.engine.core.common.businessentities.LUNs@ff187c65, org.ovirt.engine.core.common.businessentities.LUNs@2f995e1, org.ovirt.engine.core.common.businessentities.LUNs@42014f82, org.ovirt.engine.core.common.businessentities.LUNs@d50528ed], log id: 5ef4ed81 >2012-09-02 15:14:48,994 INFO [org.ovirt.engine.core.bll.storage.AddSANStorageDomainCommand] (ajp-/127.0.0.1:8009-2) [1bd3f41] Running command: AddSANStorageDomainCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2012-09-02 15:14:49,008 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVGVDSCommand] (ajp-/127.0.0.1:8009-2) [1bd3f41] START, CreateVGVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storageDomainId=8e6a506f-5b50-4438-88fd-533bc88e481e, deviceList=[gadi-lun1], force=false), log id: 69696d29 >2012-09-02 15:14:49,035 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (ajp-/127.0.0.1:8009-2) [1bd3f41] Failed in CreateVGVDS method >2012-09-02 15:14:49,035 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (ajp-/127.0.0.1:8009-2) [1bd3f41] Error code InvalidPhysDev and error message VDSGenericException: VDSErrorException: Failed to CreateVGVDS, error = Invalid physical device: ('gadi-lun1',) >2012-09-02 15:14:49,036 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (ajp-/127.0.0.1:8009-2) [1bd3f41] Command org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVGVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.irsbroker.OneUuidReturnForXmlRpc >mUuid Null >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 606 >mMessage Invalid physical device: ('gadi-lun1',) > > >2012-09-02 15:14:49,036 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (ajp-/127.0.0.1:8009-2) [1bd3f41] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:14:49,036 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (ajp-/127.0.0.1:8009-2) [1bd3f41] Command CreateVGVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to CreateVGVDS, error = Invalid physical device: ('gadi-lun1',) >2012-09-02 15:14:49,036 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVGVDSCommand] (ajp-/127.0.0.1:8009-2) [1bd3f41] FINISH, CreateVGVDSCommand, log id: 69696d29 >2012-09-02 15:14:49,036 ERROR [org.ovirt.engine.core.bll.storage.AddSANStorageDomainCommand] (ajp-/127.0.0.1:8009-2) [1bd3f41] Command org.ovirt.engine.core.bll.storage.AddSANStorageDomainCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to CreateVGVDS, error = Invalid physical device: ('gadi-lun1',) >2012-09-02 15:14:49,041 INFO [org.ovirt.engine.core.bll.storage.AddSANStorageDomainCommand] (ajp-/127.0.0.1:8009-2) [1bd3f41] Command [id=73817072-449d-4bd4-b342-13e84f9bb4b7]: Compensating NEW_ENTITY_ID of org.ovirt.engine.core.common.businessentities.storage_domain_dynamic; snapshot: 8e6a506f-5b50-4438-88fd-533bc88e481e. >2012-09-02 15:14:49,042 INFO [org.ovirt.engine.core.bll.storage.AddSANStorageDomainCommand] (ajp-/127.0.0.1:8009-2) [1bd3f41] Command [id=73817072-449d-4bd4-b342-13e84f9bb4b7]: Compensating NEW_ENTITY_ID of org.ovirt.engine.core.common.businessentities.storage_domain_static; snapshot: 8e6a506f-5b50-4438-88fd-533bc88e481e. >2012-09-02 15:14:49,060 ERROR [org.ovirt.engine.core.bll.storage.AddSANStorageDomainCommand] (ajp-/127.0.0.1:8009-2) [1bd3f41] Transaction rolled-back for command: org.ovirt.engine.core.bll.storage.AddSANStorageDomainCommand. >2012-09-02 15:14:49,077 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (ajp-/127.0.0.1:8009-2) Operation Failed: [] >2012-09-02 15:15:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-90) Autorecovering hosts is disabled, skipping >2012-09-02 15:15:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-90) Autorecovering storage domains is disabled, skipping >2012-09-02 15:15:19,208 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8009-7) Checking if user admin@internal is an admin, result true >2012-09-02 15:15:19,208 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8009-7) Running command: LoginUserCommand internal: false. >2012-09-02 15:15:19,245 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8009-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2012-09-02 15:15:19,442 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8009-6) Checking if user admin@internal is an admin, result true >2012-09-02 15:15:19,442 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8009-6) Running command: LoginUserCommand internal: false. >2012-09-02 15:15:19,511 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8009-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2012-09-02 15:15:21,223 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (ajp-/127.0.0.1:8009-5) [247853d4] START, ValidateStorageServerConnectionVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = ISCSI, connectionList = [{ id: null, connection: 10.35.64.10 };]), log id: 633bfc9d >2012-09-02 15:15:21,242 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (ajp-/127.0.0.1:8009-5) [247853d4] FINISH, ValidateStorageServerConnectionVDSCommand, return: {00000000-0000-0000-0000-000000000000=0}, log id: 633bfc9d >2012-09-02 15:15:21,270 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (ajp-/127.0.0.1:8009-5) [247853d4] Running command: ConnectStorageToVdsCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2012-09-02 15:15:21,271 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8009-5) [247853d4] START, ConnectStorageServerVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = ISCSI, connectionList = [{ id: null, connection: 10.35.64.10 };]), log id: fbcdd77 >2012-09-02 15:15:21,341 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8009-5) [247853d4] FINISH, ConnectStorageServerVDSCommand, return: {00000000-0000-0000-0000-000000000000=0}, log id: fbcdd77 >2012-09-02 15:15:21,355 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetDeviceListVDSCommand] (ajp-/127.0.0.1:8009-5) START, GetDeviceListVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storageType=ISCSI), log id: 79409038 >2012-09-02 15:15:22,402 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetDeviceListVDSCommand] (ajp-/127.0.0.1:8009-5) FINISH, GetDeviceListVDSCommand, return: [org.ovirt.engine.core.common.businessentities.LUNs@81090923, org.ovirt.engine.core.common.businessentities.LUNs@ff187c65, org.ovirt.engine.core.common.businessentities.LUNs@2f995e1, org.ovirt.engine.core.common.businessentities.LUNs@42014f82, org.ovirt.engine.core.common.businessentities.LUNs@d50528ed], log id: 79409038 >2012-09-02 15:15:22,477 INFO [org.ovirt.engine.core.bll.storage.AddSANStorageDomainCommand] (ajp-/127.0.0.1:8009-5) [3516b1d9] Running command: AddSANStorageDomainCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2012-09-02 15:15:22,512 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVGVDSCommand] (ajp-/127.0.0.1:8009-5) [3516b1d9] START, CreateVGVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storageDomainId=02d6baae-2688-4075-83e7-80b9da44b7dd, deviceList=[1gadi-lun213465860], force=false), log id: 22866ccc >2012-09-02 15:15:22,728 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (ajp-/127.0.0.1:8009-5) [3516b1d9] Failed in CreateVGVDS method >2012-09-02 15:15:22,728 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (ajp-/127.0.0.1:8009-5) [3516b1d9] Error code PhysDevInitializationError and error message VDSGenericException: VDSErrorException: Failed to CreateVGVDS, error = Failed to initialize physical device: ("['/dev/mapper/1gadi-lun213465860']",) >2012-09-02 15:15:22,728 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (ajp-/127.0.0.1:8009-5) [3516b1d9] Command org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVGVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.irsbroker.OneUuidReturnForXmlRpc >mUuid Null >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 601 >mMessage Failed to initialize physical device: ("['/dev/mapper/1gadi-lun213465860']",) > > >2012-09-02 15:15:22,728 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (ajp-/127.0.0.1:8009-5) [3516b1d9] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:15:22,728 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (ajp-/127.0.0.1:8009-5) [3516b1d9] Command CreateVGVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to CreateVGVDS, error = Failed to initialize physical device: ("['/dev/mapper/1gadi-lun213465860']",) >2012-09-02 15:15:22,728 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVGVDSCommand] (ajp-/127.0.0.1:8009-5) [3516b1d9] FINISH, CreateVGVDSCommand, log id: 22866ccc >2012-09-02 15:15:22,728 ERROR [org.ovirt.engine.core.bll.storage.AddSANStorageDomainCommand] (ajp-/127.0.0.1:8009-5) [3516b1d9] Command org.ovirt.engine.core.bll.storage.AddSANStorageDomainCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to CreateVGVDS, error = Failed to initialize physical device: ("['/dev/mapper/1gadi-lun213465860']",) >2012-09-02 15:15:22,732 INFO [org.ovirt.engine.core.bll.storage.AddSANStorageDomainCommand] (ajp-/127.0.0.1:8009-5) [3516b1d9] Command [id=213e0e57-e9e2-4327-9e7b-96c0de08d787]: Compensating NEW_ENTITY_ID of org.ovirt.engine.core.common.businessentities.storage_domain_dynamic; snapshot: 02d6baae-2688-4075-83e7-80b9da44b7dd. >2012-09-02 15:15:22,736 INFO [org.ovirt.engine.core.bll.storage.AddSANStorageDomainCommand] (ajp-/127.0.0.1:8009-5) [3516b1d9] Command [id=213e0e57-e9e2-4327-9e7b-96c0de08d787]: Compensating NEW_ENTITY_ID of org.ovirt.engine.core.common.businessentities.storage_domain_static; snapshot: 02d6baae-2688-4075-83e7-80b9da44b7dd. >2012-09-02 15:15:22,751 ERROR [org.ovirt.engine.core.bll.storage.AddSANStorageDomainCommand] (ajp-/127.0.0.1:8009-5) [3516b1d9] Transaction rolled-back for command: org.ovirt.engine.core.bll.storage.AddSANStorageDomainCommand. >2012-09-02 15:15:22,792 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (ajp-/127.0.0.1:8009-5) Operation Failed: [] >2012-09-02 15:17:20,514 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8009-11) Checking if user admin@internal is an admin, result true >2012-09-02 15:17:20,515 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8009-11) Running command: LoginUserCommand internal: false. >2012-09-02 15:17:20,602 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8009-11) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2012-09-02 15:17:20,792 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8009-7) Checking if user admin@internal is an admin, result true >2012-09-02 15:17:20,792 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8009-7) Running command: LoginUserCommand internal: false. >2012-09-02 15:17:20,826 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8009-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2012-09-02 15:17:22,548 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (ajp-/127.0.0.1:8009-9) [2a6a6234] START, ValidateStorageServerConnectionVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = ISCSI, connectionList = [{ id: null, connection: 10.35.64.10 };]), log id: 7c9b6c41 >2012-09-02 15:17:22,568 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (ajp-/127.0.0.1:8009-9) [2a6a6234] FINISH, ValidateStorageServerConnectionVDSCommand, return: {00000000-0000-0000-0000-000000000000=0}, log id: 7c9b6c41 >2012-09-02 15:17:22,588 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (ajp-/127.0.0.1:8009-9) [2a6a6234] Running command: ConnectStorageToVdsCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2012-09-02 15:17:22,589 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8009-9) [2a6a6234] START, ConnectStorageServerVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = ISCSI, connectionList = [{ id: null, connection: 10.35.64.10 };]), log id: 9578684 >2012-09-02 15:17:22,707 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8009-9) [2a6a6234] FINISH, ConnectStorageServerVDSCommand, return: {00000000-0000-0000-0000-000000000000=0}, log id: 9578684 >2012-09-02 15:17:22,734 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetDeviceListVDSCommand] (ajp-/127.0.0.1:8009-9) START, GetDeviceListVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storageType=ISCSI), log id: 23c08aad >2012-09-02 15:17:23,704 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetDeviceListVDSCommand] (ajp-/127.0.0.1:8009-9) FINISH, GetDeviceListVDSCommand, return: [org.ovirt.engine.core.common.businessentities.LUNs@ff187c65, org.ovirt.engine.core.common.businessentities.LUNs@9a09a574, org.ovirt.engine.core.common.businessentities.LUNs@81090923, org.ovirt.engine.core.common.businessentities.LUNs@42014f82, org.ovirt.engine.core.common.businessentities.LUNs@2f995e1], log id: 23c08aad >2012-09-02 15:17:23,763 INFO [org.ovirt.engine.core.bll.storage.AddSANStorageDomainCommand] (ajp-/127.0.0.1:8009-9) [1865b1a0] Running command: AddSANStorageDomainCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2012-09-02 15:17:23,779 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVGVDSCommand] (ajp-/127.0.0.1:8009-9) [1865b1a0] START, CreateVGVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storageDomainId=56a64ce1-90ab-4576-86fa-c269160aaa38, deviceList=[1gadi-lun213465860], force=false), log id: 2521c0b5 >2012-09-02 15:17:24,709 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVGVDSCommand] (ajp-/127.0.0.1:8009-9) [1865b1a0] FINISH, CreateVGVDSCommand, return: S3DGFv-BNUS-OF2a-41c3-lQWp-1sUM-nchOkj, log id: 2521c0b5 >2012-09-02 15:17:24,711 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (ajp-/127.0.0.1:8009-9) [1865b1a0] START, CreateStorageDomainVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storageDomain=org.ovirt.engine.core.common.businessentities.storage_domain_static@9f7ad26e, args=S3DGFv-BNUS-OF2a-41c3-lQWp-1sUM-nchOkj), log id: 52a77e96 >2012-09-02 15:17:30,458 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (ajp-/127.0.0.1:8009-9) [1865b1a0] FINISH, CreateStorageDomainVDSCommand, log id: 52a77e96 >2012-09-02 15:17:30,461 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStorageDomainStatsVDSCommand] (ajp-/127.0.0.1:8009-9) [1865b1a0] START, GetStorageDomainStatsVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storageDomainId=56a64ce1-90ab-4576-86fa-c269160aaa38), log id: 409c6560 >2012-09-02 15:17:30,670 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStorageDomainStatsVDSCommand] (ajp-/127.0.0.1:8009-9) [1865b1a0] FINISH, GetStorageDomainStatsVDSCommand, return: org.ovirt.engine.core.common.businessentities.storage_domains@2ef81839, log id: 409c6560 >2012-09-02 15:17:30,717 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVGInfoVDSCommand] (ajp-/127.0.0.1:8009-9) [1865b1a0] START, GetVGInfoVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, VGID=S3DGFv-BNUS-OF2a-41c3-lQWp-1sUM-nchOkj), log id: 2e842ead >2012-09-02 15:17:30,753 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVGInfoVDSCommand] (ajp-/127.0.0.1:8009-9) [1865b1a0] FINISH, GetVGInfoVDSCommand, return: [org.ovirt.engine.core.common.businessentities.LUNs@c0bce0c2], log id: 2e842ead >2012-09-02 15:17:31,188 INFO [org.ovirt.engine.core.bll.storage.AttachStorageDomainToPoolCommand] (ajp-/127.0.0.1:8009-11) [3367ce6a] Running command: AttachStorageDomainToPoolCommand internal: false. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:17:31,194 INFO [org.ovirt.engine.core.bll.storage.AddStoragePoolWithStoragesCommand] (ajp-/127.0.0.1:8009-11) [2850e1d7] Lock Acquired to object EngineLock [exclusiveLocks= key: bd560b80-c245-46b3-ad8c-b142a3460cf6 value: POOL >, sharedLocks= ] >2012-09-02 15:17:31,217 INFO [org.ovirt.engine.core.bll.storage.AddStoragePoolWithStoragesCommand] (ajp-/127.0.0.1:8009-11) [2850e1d7] Running command: AddStoragePoolWithStoragesCommand internal: true. Entities affected : ID: bd560b80-c245-46b3-ad8c-b142a3460cf6 Type: StoragePool >2012-09-02 15:17:31,332 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8009-11) [2850e1d7] START, ConnectStorageServerVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, storageType = ISCSI, connectionList = [{ id: 90cee3d3-f602-49f9-8f17-8665c22f2d06, connection: 10.35.64.10 };]), log id: 1fad1bfd >2012-09-02 15:17:31,437 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8009-11) [2850e1d7] FINISH, ConnectStorageServerVDSCommand, return: {90cee3d3-f602-49f9-8f17-8665c22f2d06=0}, log id: 1fad1bfd >2012-09-02 15:17:31,439 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStoragePoolVDSCommand] (ajp-/127.0.0.1:8009-11) [2850e1d7] START, CreateStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId=bd560b80-c245-46b3-ad8c-b142a3460cf6, storageType=ISCSI, storagePoolName=TestDataCenter, masterDomainId=56a64ce1-90ab-4576-86fa-c269160aaa38, domainsIdList=[56a64ce1-90ab-4576-86fa-c269160aaa38], masterVersion=1), log id: 371d435a >2012-09-02 15:17:58,747 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStoragePoolVDSCommand] (ajp-/127.0.0.1:8009-11) [2850e1d7] FINISH, CreateStoragePoolVDSCommand, log id: 371d435a >2012-09-02 15:17:58,761 INFO [org.ovirt.engine.core.bll.storage.AddStoragePoolWithStoragesCommand] (ajp-/127.0.0.1:8009-11) [2850e1d7] Lock freed to object EngineLock [exclusiveLocks= key: bd560b80-c245-46b3-ad8c-b142a3460cf6 value: POOL >, sharedLocks= ] >2012-09-02 15:17:58,764 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp-/127.0.0.1:8009-11) [496de3ca] Lock Acquired to object EngineLock [exclusiveLocks= key: 56a64ce1-90ab-4576-86fa-c269160aaa38 value: STORAGE >, sharedLocks= ] >2012-09-02 15:17:58,781 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp-/127.0.0.1:8009-11) [496de3ca] Running command: ActivateStorageDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:17:58,804 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp-/127.0.0.1:8009-11) [496de3ca] Lock freed to object EngineLock [exclusiveLocks= key: 56a64ce1-90ab-4576-86fa-c269160aaa38 value: STORAGE >, sharedLocks= ] >2012-09-02 15:17:58,804 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp-/127.0.0.1:8009-11) [496de3ca] ActivateStorage Domain. Before Connect all hosts to pool. Time:9/2/12 3:17 PM >2012-09-02 15:17:58,817 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-4-thread-48) [496de3ca] START, ConnectStorageServerVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, storageType = ISCSI, connectionList = [{ id: 90cee3d3-f602-49f9-8f17-8665c22f2d06, connection: 10.35.64.10 };]), log id: 2bb103d3 >2012-09-02 15:17:58,917 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-4-thread-48) [496de3ca] FINISH, ConnectStorageServerVDSCommand, return: {90cee3d3-f602-49f9-8f17-8665c22f2d06=0}, log id: 2bb103d3 >2012-09-02 15:17:58,917 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp-/127.0.0.1:8009-11) [496de3ca] ActivateStorage Domain. After Connect all hosts to pool. Time:9/2/12 3:17 PM >2012-09-02 15:17:58,918 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (ajp-/127.0.0.1:8009-11) [496de3ca] START, ActivateStorageDomainVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, storageDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38), log id: 6b1191cd >2012-09-02 15:17:58,922 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (ajp-/127.0.0.1:8009-11) [10f79652] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: bd560b80-c245-46b3-ad8c-b142a3460cf6 Type: StoragePool >2012-09-02 15:17:58,984 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (ajp-/127.0.0.1:8009-11) [10f79652] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:17:59,167 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (ajp-/127.0.0.1:8009-11) [10f79652] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: e6ee2a7 >2012-09-02 15:18:01,624 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (ajp-/127.0.0.1:8009-11) [10f79652] FINISH, ConnectStoragePoolVDSCommand, log id: e6ee2a7 >2012-09-02 15:18:01,643 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (ajp-/127.0.0.1:8009-11) [10f79652] hostFromVds::Connected host to pool - selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Free, storage pool TestDataCenter >2012-09-02 15:18:01,677 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (ajp-/127.0.0.1:8009-11) [10f79652] starting spm on vds green-vdsa.qa.lab.tlv.redhat.com, storage pool TestDataCenter, prevId -1, LVER -1 >2012-09-02 15:18:01,708 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (ajp-/127.0.0.1:8009-11) [10f79652] START, SpmStartVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 47969aae >2012-09-02 15:18:01,730 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (ajp-/127.0.0.1:8009-11) [10f79652] spmStart polling started: taskId = a203ab9c-af8c-4da4-8249-780b24316571 >2012-09-02 15:18:24,369 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (ajp-/127.0.0.1:8009-11) [10f79652] spmStart polling ended: taskId = a203ab9c-af8c-4da4-8249-780b24316571 task status = finished >2012-09-02 15:18:24,387 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (ajp-/127.0.0.1:8009-11) [10f79652] spmStart polling ended. spm status: SPM >2012-09-02 15:18:24,388 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (ajp-/127.0.0.1:8009-11) [10f79652] START, HSMClearTaskVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, taskId=a203ab9c-af8c-4da4-8249-780b24316571), log id: 61606d76 >2012-09-02 15:18:24,407 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (ajp-/127.0.0.1:8009-11) [10f79652] FINISH, HSMClearTaskVDSCommand, log id: 61606d76 >2012-09-02 15:18:24,407 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (ajp-/127.0.0.1:8009-11) [10f79652] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1b28fec, log id: 47969aae >2012-09-02 15:18:24,415 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (ajp-/127.0.0.1:8009-11) [10f79652] Initialize Irs proxy from vds: 10.35.102.10 >2012-09-02 15:18:24,482 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (ajp-/127.0.0.1:8009-11) [10f79652] FINISH, ActivateStorageDomainVDSCommand, log id: 6b1191cd >2012-09-02 15:18:24,482 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp-/127.0.0.1:8009-11) [10f79652] ActivateStorage Domain. After Activate storage domain in vds. Time:9/2/12 3:18 PM >2012-09-02 15:18:24,514 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-47) [10f79652] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 11dd9e4c >2012-09-02 15:18:24,544 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-47) [10f79652] FINISH, ConnectStoragePoolVDSCommand, log id: 11dd9e4c >2012-09-02 15:18:24,544 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp-/127.0.0.1:8009-11) [10f79652] ActivateStorage Domain. After Refresh all pools . Time:9/2/12 3:18 PM >2012-09-02 15:18:24,601 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp-/127.0.0.1:8009-11) [10f79652] ActivateStorage Domain. After change storage pool status in vds. Time:9/2/12 3:18 PM >2012-09-02 15:18:25,693 INFO [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (ajp-/127.0.0.1:8009-11) [5a3800eb] Lock Acquired to object EngineLock [exclusiveLocks= key: testVm value: VM_NAME >, sharedLocks= ] >2012-09-02 15:18:25,733 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8009-11) [5a3800eb] START, IsValidVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 9dc8d13 >2012-09-02 15:18:25,734 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8009-11) [5a3800eb] FINISH, IsValidVDSCommand, return: true, log id: 9dc8d13 >2012-09-02 15:18:25,762 INFO [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (ajp-/127.0.0.1:8009-11) [5a3800eb] Running command: AddVmFromScratchCommand internal: false. Entities affected : ID: b1600626-f4e9-11e1-b9cc-001a4a169705 Type: VdsGroups >2012-09-02 15:18:25,763 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8009-11) [5a3800eb] START, IsValidVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 79cb745f >2012-09-02 15:18:25,763 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8009-11) [5a3800eb] FINISH, IsValidVDSCommand, return: true, log id: 79cb745f >2012-09-02 15:18:25,831 INFO [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (ajp-/127.0.0.1:8009-11) [5a3800eb] Lock freed to object EngineLock [exclusiveLocks= key: testVm value: VM_NAME >, sharedLocks= ] >2012-09-02 15:18:26,343 INFO [org.ovirt.engine.core.bll.AddVmInterfaceCommand] (ajp-/127.0.0.1:8009-1) [57f862ff] Running command: AddVmInterfaceCommand internal: false. Entities affected : ID: 5449d6cf-0e31-44b5-b0a9-65036f122662 Type: VM >2012-09-02 15:18:26,345 INFO [org.ovirt.engine.core.bll.MacPoolManager] (ajp-/127.0.0.1:8009-1) [57f862ff] MacPoolManager::allocateNewMac entered >2012-09-02 15:18:26,345 INFO [org.ovirt.engine.core.bll.MacPoolManager] (ajp-/127.0.0.1:8009-1) [57f862ff] MacPoolManager::allocateNewMac allocated mac = 00:1a:4a:23:61:68 >2012-09-02 15:18:26,774 INFO [org.ovirt.engine.core.bll.AddDiskCommand] (ajp-/127.0.0.1:8009-11) [576f6b49] Lock Acquired to object EngineLock [exclusiveLocks= key: 5449d6cf-0e31-44b5-b0a9-65036f122662 value: VM_DISK_BOOT >, sharedLocks= key: 5449d6cf-0e31-44b5-b0a9-65036f122662 value: VM >] >2012-09-02 15:18:26,784 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8009-11) [576f6b49] START, IsValidVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 2350045a >2012-09-02 15:18:26,784 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8009-11) [576f6b49] FINISH, IsValidVDSCommand, return: true, log id: 2350045a >2012-09-02 15:18:26,809 INFO [org.ovirt.engine.core.bll.AddDiskCommand] (ajp-/127.0.0.1:8009-11) [576f6b49] Running command: AddDiskCommand internal: false. Entities affected : ID: 5449d6cf-0e31-44b5-b0a9-65036f122662 Type: VM, ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:18:26,809 INFO [org.ovirt.engine.core.bll.ImagesHandler] (ajp-/127.0.0.1:8009-11) [576f6b49] Disk alias retrieved from the client is null or empty, the suggested default disk alias to be used is testVm_Disk1 >2012-09-02 15:18:26,956 INFO [org.ovirt.engine.core.bll.AddImageFromScratchCommand] (ajp-/127.0.0.1:8009-11) [61e6cf68] Running command: AddImageFromScratchCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:18:27,010 INFO [org.ovirt.engine.core.bll.AddImageFromScratchCommand] (ajp-/127.0.0.1:8009-11) [61e6cf68] Lock freed to object EngineLock [exclusiveLocks= key: 5449d6cf-0e31-44b5-b0a9-65036f122662 value: VM_DISK_BOOT >, sharedLocks= key: 5449d6cf-0e31-44b5-b0a9-65036f122662 value: VM >] >2012-09-02 15:18:27,013 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (ajp-/127.0.0.1:8009-11) [61e6cf68] START, CreateImageVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = 3.1, storageDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, imageGroupId = 3418064e-f30d-4fdb-b00b-3057eded62b4, imageSizeInBytes = 2147483648, volumeFormat = COW, newImageId = b40bb342-3a13-45af-b862-d809bfb05d4d, newImageDescription = ), log id: 7a4a0c86 >2012-09-02 15:18:27,013 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (ajp-/127.0.0.1:8009-11) [61e6cf68] -- CreateImageVDSCommand::ExecuteIrsBrokerCommand: calling 'createVolume' with two new parameters: description and UUID >2012-09-02 15:18:27,013 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (ajp-/127.0.0.1:8009-11) [61e6cf68] -- createVolume parameters: > sdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38 > spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6 > imgGUID=3418064e-f30d-4fdb-b00b-3057eded62b4 > size=2,147,483,648 bytes > volFormat=COW > volType=Sparse > volUUID=b40bb342-3a13-45af-b862-d809bfb05d4d > descr= > srcImgGUID=00000000-0000-0000-0000-000000000000 > srcVolUUID=00000000-0000-0000-0000-000000000000 > >2012-09-02 15:18:27,775 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (ajp-/127.0.0.1:8009-11) [61e6cf68] FINISH, CreateImageVDSCommand, return: b40bb342-3a13-45af-b862-d809bfb05d4d, log id: 7a4a0c86 >2012-09-02 15:18:27,788 INFO [org.ovirt.engine.core.bll.EntityAsyncTask] (ajp-/127.0.0.1:8009-11) [61e6cf68] EntityAsyncTask::Adding EntityMultiAsyncTasks object for entity 3418064e-f30d-4fdb-b00b-3057eded62b4 >2012-09-02 15:18:27,789 INFO [org.ovirt.engine.core.bll.EntityMultiAsyncTasks] (ajp-/127.0.0.1:8009-11) [61e6cf68] EntityMultiAsyncTasks::AttachTask: Attaching task 17201314-d524-4ac2-98d9-8c886064445f to entity 3418064e-f30d-4fdb-b00b-3057eded62b4. >2012-09-02 15:18:27,802 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskUtils] (ajp-/127.0.0.1:8009-11) [61e6cf68] Adding task 17201314-d524-4ac2-98d9-8c886064445f to DataBase >2012-09-02 15:18:27,893 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (ajp-/127.0.0.1:8009-11) [61e6cf68] Adding task 17201314-d524-4ac2-98d9-8c886064445f (Parent Command AddDisk, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters), polling hasn't started yet.. >2012-09-02 15:18:28,014 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (ajp-/127.0.0.1:8009-11) [61e6cf68] BaseAsyncTask::StartPollingTask: Starting to poll task 17201314-d524-4ac2-98d9-8c886064445f. >2012-09-02 15:18:28,014 INFO [org.ovirt.engine.core.bll.EntityMultiAsyncTasks] (ajp-/127.0.0.1:8009-11) [61e6cf68] EntityMultiAsyncTasks::StartPollingTask: Current Action Type for entity 3418064e-f30d-4fdb-b00b-3057eded62b4 is AddDisk (determined by task 17201314-d524-4ac2-98d9-8c886064445f) >2012-09-02 15:18:34,991 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-75) Polling and updating Async Tasks: 1 tasks, 1 tasks to poll now >2012-09-02 15:18:35,020 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (QuartzScheduler_Worker-75) SPMAsyncTask::PollTask: Polling task 17201314-d524-4ac2-98d9-8c886064445f (Parent Command AddDisk, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status finished, result 'success'. >2012-09-02 15:18:35,043 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (QuartzScheduler_Worker-75) BaseAsyncTask::OnTaskEndSuccess: Task 17201314-d524-4ac2-98d9-8c886064445f (Parent Command AddDisk, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended successfully. >2012-09-02 15:18:35,044 INFO [org.ovirt.engine.core.bll.EntityAsyncTask] (QuartzScheduler_Worker-75) EntityAsyncTask::EndActionIfNecessary: All tasks of entity 3418064e-f30d-4fdb-b00b-3057eded62b4 has ended -> executing EndAction >2012-09-02 15:18:35,044 INFO [org.ovirt.engine.core.bll.EntityAsyncTask] (QuartzScheduler_Worker-75) EntityAsyncTask::EndAction: Ending action for 1 tasks (entity ID: 3418064e-f30d-4fdb-b00b-3057eded62b4): calling EndAction for action type AddDisk. >2012-09-02 15:18:35,047 INFO [org.ovirt.engine.core.bll.EntityAsyncTask] (pool-4-thread-47) EntityAsyncTask::EndCommandAction [within thread]context: Attempting to EndAction AddDisk >2012-09-02 15:18:35,127 INFO [org.ovirt.engine.core.bll.AddDiskCommand] (pool-4-thread-47) [576f6b49] Ending command successfully: org.ovirt.engine.core.bll.AddDiskCommand >2012-09-02 15:18:35,127 INFO [org.ovirt.engine.core.bll.AddImageFromScratchCommand] (pool-4-thread-47) [61e6cf68] Ending command successfully: org.ovirt.engine.core.bll.AddImageFromScratchCommand >2012-09-02 15:18:35,141 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (pool-4-thread-47) [61e6cf68] START, GetImageInfoVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, storageDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, imageGroupId = 3418064e-f30d-4fdb-b00b-3057eded62b4, imageId = b40bb342-3a13-45af-b862-d809bfb05d4d), log id: b4babbf >2012-09-02 15:18:35,554 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (pool-4-thread-47) [61e6cf68] FINISH, GetImageInfoVDSCommand, return: org.ovirt.engine.core.common.businessentities.DiskImage@a2824753, log id: b4babbf >2012-09-02 15:18:35,728 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UpdateVMVDSCommand] (pool-4-thread-47) START, UpdateVMVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, storageDomainId = 00000000-0000-0000-0000-000000000000, infoDictionary.size = 1), log id: 53556bf2 >2012-09-02 15:18:35,776 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UpdateVMVDSCommand] (pool-4-thread-47) FINISH, UpdateVMVDSCommand, log id: 53556bf2 >2012-09-02 15:18:36,488 INFO [org.ovirt.engine.core.bll.EntityAsyncTask] (pool-4-thread-47) EntityAsyncTask::HandleEndActionResult [within thread]: EndAction for action type AddDisk completed, handling the result. >2012-09-02 15:18:36,489 INFO [org.ovirt.engine.core.bll.EntityAsyncTask] (pool-4-thread-47) EntityAsyncTask::HandleEndActionResult [within thread]: EndAction for action type AddDisk succeeded, clearing tasks. >2012-09-02 15:18:36,504 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (pool-4-thread-47) SPMAsyncTask::ClearAsyncTask: Attempting to clear task 17201314-d524-4ac2-98d9-8c886064445f >2012-09-02 15:18:36,505 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (pool-4-thread-47) START, SPMClearTaskVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, taskId = 17201314-d524-4ac2-98d9-8c886064445f), log id: 2548896e >2012-09-02 15:18:36,507 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (pool-4-thread-47) START, HSMClearTaskVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, taskId=17201314-d524-4ac2-98d9-8c886064445f), log id: 2c374e6f >2012-09-02 15:18:36,539 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (pool-4-thread-47) FINISH, HSMClearTaskVDSCommand, log id: 2c374e6f >2012-09-02 15:18:36,539 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (pool-4-thread-47) FINISH, SPMClearTaskVDSCommand, log id: 2548896e >2012-09-02 15:18:36,567 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (pool-4-thread-47) BaseAsyncTask::RemoveTaskFromDB: Removed task 17201314-d524-4ac2-98d9-8c886064445f from DataBase >2012-09-02 15:18:36,567 INFO [org.ovirt.engine.core.bll.EntityAsyncTask] (pool-4-thread-47) EntityAsyncTask::HandleEndActionResult [within thread]: Removing EntityMultiAsyncTasks object for entity 3418064e-f30d-4fdb-b00b-3057eded62b4 >2012-09-02 15:18:38,444 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8009-11) [57b58682] START, IsValidVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 1254526 >2012-09-02 15:18:38,449 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8009-11) [57b58682] FINISH, IsValidVDSCommand, return: true, log id: 1254526 >2012-09-02 15:18:38,475 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp-/127.0.0.1:8009-11) [57b58682] START, IsVmDuringInitiatingVDSCommand(vmId = 5449d6cf-0e31-44b5-b0a9-65036f122662), log id: 1b727f7b >2012-09-02 15:18:38,476 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp-/127.0.0.1:8009-11) [57b58682] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 1b727f7b >2012-09-02 15:18:38,534 INFO [org.ovirt.engine.core.bll.RunVmOnceCommand] (ajp-/127.0.0.1:8009-11) [57b58682] Running command: RunVmOnceCommand internal: false. Entities affected : ID: 5449d6cf-0e31-44b5-b0a9-65036f122662 Type: VM >2012-09-02 15:18:38,567 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8009-11) [57b58682] START, CreateVmVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, vmId=5449d6cf-0e31-44b5-b0a9-65036f122662, vm=org.ovirt.engine.core.common.businessentities.VM@7d60c7fb), log id: 2f27adf >2012-09-02 15:18:38,593 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8009-11) [57b58682] START, CreateVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, vmId=5449d6cf-0e31-44b5-b0a9-65036f122662, vm=org.ovirt.engine.core.common.businessentities.VM@7d60c7fb), log id: 3274bc04 >2012-09-02 15:18:38,725 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8009-11) [57b58682] org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand spiceSslCipherSuite=DEFAULT,memSize=1024,kvmEnable=true,smp=1,emulatedMachine=rhel6.3.0,vmType=kvm,keyboardLayout=en-us,nice=0,display=qxl,smpCoresPerSocket=1,spiceSecureChannels=smain,sinputs,displayNetwork=rhevm,timeOffset=0,transparentHugePages=true,vmId=5449d6cf-0e31-44b5-b0a9-65036f122662,devices=[Ljava.util.Map;@3041c3c3,acpiEnable=true,vmName=testVm,cpuType=Conroe,custom={} >2012-09-02 15:18:38,725 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8009-11) [57b58682] FINISH, CreateVDSCommand, log id: 3274bc04 >2012-09-02 15:18:38,734 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8009-11) [57b58682] IncreasePendingVms::CreateVmIncreasing vds green-vdsa.qa.lab.tlv.redhat.com pending vcpu count, now 1. Vm: testVm >2012-09-02 15:18:38,760 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8009-11) [57b58682] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 2f27adf >2012-09-02 15:18:42,437 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-89) VM testVm 5449d6cf-0e31-44b5-b0a9-65036f122662 moved from WaitForLaunch --> PoweringUp >2012-09-02 15:18:42,462 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (QuartzScheduler_Worker-89) START, FullListVdsCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, vds=null, vmIds=[5449d6cf-0e31-44b5-b0a9-65036f122662]), log id: 6204acfd >2012-09-02 15:18:42,494 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (QuartzScheduler_Worker-89) FINISH, FullListVdsCommand, return: [Lorg.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcStruct;@20f94851, log id: 6204acfd >2012-09-02 15:19:39,872 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-63) VM testVm 5449d6cf-0e31-44b5-b0a9-65036f122662 moved from PoweringUp --> Up >2012-09-02 15:19:40,222 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsDynamicDataVDSCommand] (QuartzScheduler_Worker-63) START, UpdateVdsDynamicDataVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, vdsDynamic=org.ovirt.engine.core.common.businessentities.VdsDynamic@873fd66e), log id: 5d811d8e >2012-09-02 15:19:40,255 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsDynamicDataVDSCommand] (QuartzScheduler_Worker-63) FINISH, UpdateVdsDynamicDataVDSCommand, log id: 5d811d8e >2012-09-02 15:19:42,670 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-64) Setting new tasks map. The map contains now 0 tasks >2012-09-02 15:19:42,670 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-64) Cleared all tasks of pool bd560b80-c245-46b3-ad8c-b142a3460cf6. >2012-09-02 15:19:42,674 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-64) Could not find vds that is spm and non-operational. >2012-09-02 15:20:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-96) Autorecovering hosts is disabled, skipping >2012-09-02 15:20:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-96) Autorecovering storage domains is disabled, skipping >2012-09-02 15:21:02,666 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8009-5) Checking if user admin@internal is an admin, result true >2012-09-02 15:21:02,666 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8009-5) Running command: LoginUserCommand internal: false. >2012-09-02 15:21:02,692 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8009-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2012-09-02 15:21:03,319 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8009-7) Checking if user admin@internal is an admin, result true >2012-09-02 15:21:03,320 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8009-7) Running command: LoginUserCommand internal: false. >2012-09-02 15:21:03,368 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8009-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2012-09-02 15:21:42,231 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:22:05,284 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-75) ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 15:22:07,293 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-70) ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 15:22:09,303 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-74) [59641388] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 15:22:10,861 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-77) [1b2dfbbc] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: bd560b80-c245-46b3-ad8c-b142a3460cf6 Type: StoragePool >2012-09-02 15:22:10,948 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [1b2dfbbc] IrsBroker::Failed::GetStoragePoolInfoVDS due to: ConnectException: Connection refused >2012-09-02 15:22:11,311 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-84) ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 15:22:13,323 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-83) [28a39dff] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 15:22:15,471 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-88) ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 15:22:17,478 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-95) ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 15:22:19,488 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-93) ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 15:22:20,962 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) IrsBroker::Failed::GetStoragePoolInfoVDS due to: ConnectException: Connection refused >2012-09-02 15:22:21,496 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-98) [5bec48ce] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 15:22:23,503 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-78) ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 15:22:25,511 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-1) ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 15:22:27,520 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-8) ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 15:22:29,539 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-6) ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 15:22:30,973 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) IrsBroker::Failed::GetStoragePoolInfoVDS due to: ConnectException: Connection refused >2012-09-02 15:22:31,549 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-9) ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 15:22:33,558 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-15) ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 15:22:35,565 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-18) ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 15:22:37,573 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-23) ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 15:22:39,583 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-19) ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 15:22:40,985 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) IrsBroker::Failed::GetStoragePoolInfoVDS due to: ConnectException: Connection refused >2012-09-02 15:22:41,594 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-24) ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 15:22:43,606 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-22) ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 15:22:45,618 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-35) [372f1ff9] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 15:22:47,625 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-32) ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 15:22:49,636 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-39) ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 15:22:50,997 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-37) [36253582] IrsBroker::Failed::GetStoragePoolInfoVDS due to: ConnectException: Connection refused >2012-09-02 15:22:51,644 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-42) ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 15:22:53,652 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-43) ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 15:22:55,662 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-44) ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 15:22:57,674 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-48) ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 15:22:59,688 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-52) ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 15:23:01,005 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-53) IrsBroker::Failed::GetStoragePoolInfoVDS due to: ConnectException: Connection refused >2012-09-02 15:23:01,701 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-56) ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 15:23:03,713 ERROR [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-55) VDS::handleNetworkException Server failed to respond, vds_id = b32eacf0-f4e9-11e1-973c-001a4a169705, vds_name = green-vdsa.qa.lab.tlv.redhat.com, error = VDSNetworkException: >2012-09-02 15:23:03,729 INFO [org.ovirt.engine.core.bll.VdsEventListener] (pool-4-thread-48) ResourceManager::vdsNotResponding entered for Host b32eacf0-f4e9-11e1-973c-001a4a169705, 10.35.102.10 >2012-09-02 15:23:03,767 WARN [org.ovirt.engine.core.bll.VdsNotRespondingTreatmentCommand] (pool-4-thread-48) [522c7a47] CanDoAction of action VdsNotRespondingTreatment failed. Reasons:VDS_FENCING_DISABLED >2012-09-02 15:23:11,016 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) IrsBroker::Failed::GetStoragePoolInfoVDS due to: ConnectException: Connection refused >2012-09-02 15:23:16,284 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-80) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@11be424c >2012-09-02 15:23:16,284 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-80) Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:23:16,284 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-80) Command GetCapabilitiesVDS execution failed. Error: VDSRecoveringException: Recovering from crash or Initializing >2012-09-02 15:23:16,699 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-80) ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, error = Recovering from crash or Initializing, continuing. >2012-09-02 15:23:19,359 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-74) [59641388] Cancelling the recovery from crash timer for VDS b32eacf0-f4e9-11e1-973c-001a4a169705 because vds started initializing >2012-09-02 15:23:19,361 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-74) [3e94068] Running command: InitVdsOnUpCommand internal: true. >2012-09-02 15:23:19,422 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-74) [322beea0] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: bd560b80-c245-46b3-ad8c-b142a3460cf6 Type: StoragePool >2012-09-02 15:23:19,428 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (QuartzScheduler_Worker-74) [322beea0] START, ConnectStorageServerVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, storageType = ISCSI, connectionList = [{ id: 90cee3d3-f602-49f9-8f17-8665c22f2d06, connection: 10.35.64.10 };]), log id: 79749552 >2012-09-02 15:23:19,546 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (QuartzScheduler_Worker-74) [322beea0] FINISH, ConnectStorageServerVDSCommand, return: {90cee3d3-f602-49f9-8f17-8665c22f2d06=0}, log id: 79749552 >2012-09-02 15:23:19,546 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-74) [322beea0] Host green-vdsa.qa.lab.tlv.redhat.com storage connection was succeeded >2012-09-02 15:23:19,565 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-74) [322beea0] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 469c712 >2012-09-02 15:23:20,405 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-74) [322beea0] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:23:20,405 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-74) [322beea0] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:23:20,405 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-74) [322beea0] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:23:20,405 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-74) [322beea0] FINISH, ConnectStoragePoolVDSCommand, log id: 469c712 >2012-09-02 15:23:20,405 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-74) [322beea0] Could not connect host green-vdsa.qa.lab.tlv.redhat.com to pool TestDataCenter >2012-09-02 15:23:20,439 INFO [org.ovirt.engine.core.bll.MultipleActionsRunner] (pool-4-thread-47) [322beea0] MultipleActionsRunner of type MigrateVmToServer invoked with no actions >2012-09-02 15:23:20,451 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-74) [3628a33b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: b32eacf0-f4e9-11e1-973c-001a4a169705 Type: VDS >2012-09-02 15:23:20,464 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-74) [12d00b82] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: b32eacf0-f4e9-11e1-973c-001a4a169705 Type: VDS >2012-09-02 15:23:21,094 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [1b2dfbbc] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:23:21,119 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-77) [1b2dfbbc] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 3a40ca55 >2012-09-02 15:23:21,932 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-77) [1b2dfbbc] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:23:21,932 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-77) [1b2dfbbc] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:23:21,932 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-77) [1b2dfbbc] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:23:21,932 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-77) [1b2dfbbc] FINISH, ConnectStoragePoolVDSCommand, log id: 3a40ca55 >2012-09-02 15:23:21,932 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [1b2dfbbc] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:23:21,932 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [1b2dfbbc] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:23:22,042 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-77) [46fa81bd] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:23:22,063 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-77) [46fa81bd] START, ResetIrsVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = 10.35.102.10, vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, ignoreStopFailed = true), log id: 1cd4b121 >2012-09-02 15:23:22,064 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-77) [46fa81bd] START, SpmStopVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6), log id: 76c0c32c >2012-09-02 15:23:22,084 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-77) [46fa81bd] SpmStopVDSCommand::Stopping SPM on vds green-vdsa.qa.lab.tlv.redhat.com, pool id bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:23:22,108 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-77) [46fa81bd] FINISH, SpmStopVDSCommand, log id: 76c0c32c >2012-09-02 15:23:22,117 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-77) [46fa81bd] FINISH, ResetIrsVDSCommand, log id: 1cd4b121 >2012-09-02 15:23:22,128 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-77) [46fa81bd] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 193fa483 >2012-09-02 15:23:22,128 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [46fa81bd] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:23:22,128 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-77) [46fa81bd] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 193fa483 >2012-09-02 15:23:26,636 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-88) domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:23:32,172 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:23:32,188 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-94) START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 29f4f548 >2012-09-02 15:23:33,022 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-94) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:23:33,022 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-94) Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:23:33,022 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-94) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:23:33,022 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-94) FINISH, ConnectStoragePoolVDSCommand, log id: 29f4f548 >2012-09-02 15:23:33,022 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:23:33,022 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:23:33,114 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-94) [536973ff] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:23:33,149 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-94) [536973ff] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 654d701f >2012-09-02 15:23:33,149 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [536973ff] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:23:33,149 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-94) [536973ff] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 654d701f >2012-09-02 15:23:36,902 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:23:43,205 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-12) hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:23:43,501 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-12) START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 7f8525f6 >2012-09-02 15:23:44,336 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-12) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:23:44,336 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-12) Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:23:44,336 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-12) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:23:44,336 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-12) FINISH, ConnectStoragePoolVDSCommand, log id: 7f8525f6 >2012-09-02 15:23:44,336 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-12) IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:23:44,336 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-12) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:23:44,489 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-12) [2b77d958] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:23:44,534 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-12) [2b77d958] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3cb7c0bc >2012-09-02 15:23:44,534 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-12) [2b77d958] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:23:44,534 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-12) [2b77d958] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3cb7c0bc >2012-09-02 15:23:47,280 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-19) domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:23:54,592 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:23:54,610 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-31) START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 7dc50461 >2012-09-02 15:23:55,445 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-31) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:23:55,446 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-31) Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:23:55,446 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-31) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:23:55,446 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-31) FINISH, ConnectStoragePoolVDSCommand, log id: 7dc50461 >2012-09-02 15:23:55,446 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:23:55,446 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:23:55,519 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-31) [3fd0f791] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:23:55,552 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-31) [3fd0f791] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 61b50a14 >2012-09-02 15:23:55,552 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) [3fd0f791] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:23:55,552 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-31) [3fd0f791] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 61b50a14 >2012-09-02 15:23:57,528 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-36) domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:24:05,611 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-44) hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:24:05,630 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-44) START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 1af6eb64 >2012-09-02 15:24:06,461 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-44) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:24:06,461 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-44) Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:24:06,461 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-44) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:24:06,461 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-44) FINISH, ConnectStoragePoolVDSCommand, log id: 1af6eb64 >2012-09-02 15:24:06,461 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-44) IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:24:06,461 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-44) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:24:06,556 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-44) [7f514ddd] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:24:06,594 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-44) [7f514ddd] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 325be83f >2012-09-02 15:24:06,594 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-44) [7f514ddd] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:24:06,594 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-44) [7f514ddd] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 325be83f >2012-09-02 15:24:07,860 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-49) domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:24:16,633 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:24:16,647 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-57) START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 39f7508e >2012-09-02 15:24:17,481 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-57) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:24:17,481 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-57) Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:24:17,481 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-57) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:24:17,481 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-57) FINISH, ConnectStoragePoolVDSCommand, log id: 39f7508e >2012-09-02 15:24:17,481 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:24:17,481 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:24:17,696 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-57) [330d4585] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:24:17,735 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-57) [330d4585] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 290389b3 >2012-09-02 15:24:17,735 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [330d4585] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:24:17,735 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-57) [330d4585] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 290389b3 >2012-09-02 15:24:18,582 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [1138eff2] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:24:27,795 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-81) hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:24:27,817 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-81) START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 5a62d794 >2012-09-02 15:24:28,657 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-81) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:24:28,657 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-81) Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:24:28,657 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-81) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:24:28,657 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-81) FINISH, ConnectStoragePoolVDSCommand, log id: 5a62d794 >2012-09-02 15:24:28,657 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-81) IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:24:28,657 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-81) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:24:28,771 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-81) [6454b998] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:24:28,875 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-81) [6454b998] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 72df54e1 >2012-09-02 15:24:28,875 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-81) [6454b998] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:24:28,875 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-81) [6454b998] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 72df54e1 >2012-09-02 15:24:38,912 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-95) hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:24:38,928 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-95) START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 7377c1d4 >2012-09-02 15:24:39,091 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-92) domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:24:39,771 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-95) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:24:39,771 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-95) Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:24:39,771 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-95) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:24:39,771 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-95) FINISH, ConnectStoragePoolVDSCommand, log id: 7377c1d4 >2012-09-02 15:24:39,771 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-95) IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:24:39,771 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-95) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:24:40,216 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-95) [651f330f] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:24:40,251 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-95) [651f330f] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3573aa7b >2012-09-02 15:24:40,251 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-95) [651f330f] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:24:40,251 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-95) [651f330f] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3573aa7b >2012-09-02 15:24:49,338 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-6) domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:24:50,301 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:24:50,352 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-10) START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 6b32819c >2012-09-02 15:24:51,201 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-10) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:24:51,201 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-10) Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:24:51,201 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-10) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:24:51,201 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-10) FINISH, ConnectStoragePoolVDSCommand, log id: 6b32819c >2012-09-02 15:24:51,201 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:24:51,201 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:24:51,287 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-10) [2b98e94d] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:24:51,338 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-10) [2b98e94d] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 410c0653 >2012-09-02 15:24:51,339 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) [2b98e94d] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:24:51,339 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-10) [2b98e94d] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 410c0653 >2012-09-02 15:24:59,578 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-19) domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:25:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-25) Autorecovering hosts is disabled, skipping >2012-09-02 15:25:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-25) Autorecovering storage domains is disabled, skipping >2012-09-02 15:25:01,405 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-24) hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:25:01,429 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-24) START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 5b6d73f6 >2012-09-02 15:25:02,242 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-24) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:25:02,242 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-24) Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:25:02,242 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-24) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:25:02,242 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-24) FINISH, ConnectStoragePoolVDSCommand, log id: 5b6d73f6 >2012-09-02 15:25:02,242 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-24) IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:25:02,242 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-24) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:25:02,328 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-24) [392574c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:25:02,361 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-24) [392574c] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4ff84b8e >2012-09-02 15:25:02,361 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-24) [392574c] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:25:02,361 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-24) [392574c] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4ff84b8e >2012-09-02 15:25:10,227 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:25:12,410 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:25:12,429 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-42) START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 62297a00 >2012-09-02 15:25:13,272 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-42) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:25:13,272 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-42) Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:25:13,272 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-42) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:25:13,272 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-42) FINISH, ConnectStoragePoolVDSCommand, log id: 62297a00 >2012-09-02 15:25:13,272 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:25:13,272 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:25:13,518 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-42) [48816517] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:25:13,577 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-42) [48816517] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 30fbc140 >2012-09-02 15:25:13,577 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [48816517] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:25:13,577 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-42) [48816517] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 30fbc140 >2012-09-02 15:25:20,518 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:25:23,667 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:25:23,698 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-55) START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 24bfbe66 >2012-09-02 15:25:24,506 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-55) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:25:24,506 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-55) Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:25:24,506 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-55) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:25:24,506 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-55) FINISH, ConnectStoragePoolVDSCommand, log id: 24bfbe66 >2012-09-02 15:25:24,506 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:25:24,507 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:25:24,613 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-55) [594f6f51] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:25:24,663 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-55) [594f6f51] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 772b956e >2012-09-02 15:25:24,664 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) [594f6f51] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:25:24,664 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-55) [594f6f51] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 772b956e >2012-09-02 15:25:30,742 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:25:34,852 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:25:34,955 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-69) START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 108c8103 >2012-09-02 15:25:35,761 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-69) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:25:35,761 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-69) Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:25:35,761 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-69) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:25:35,761 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-69) FINISH, ConnectStoragePoolVDSCommand, log id: 108c8103 >2012-09-02 15:25:35,761 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:25:35,761 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:25:36,329 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-69) [27a95d66] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:25:36,374 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-69) [27a95d66] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 654fae8e >2012-09-02 15:25:36,374 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [27a95d66] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:25:36,374 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-69) [27a95d66] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 654fae8e >2012-09-02 15:25:40,952 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-76) domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:25:46,432 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:25:46,463 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-87) START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 68aaf81e >2012-09-02 15:25:47,302 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-87) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:25:47,302 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-87) Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:25:47,302 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-87) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:25:47,302 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-87) FINISH, ConnectStoragePoolVDSCommand, log id: 68aaf81e >2012-09-02 15:25:47,302 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:25:47,302 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:25:47,437 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-87) [100a6607] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:25:47,476 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-87) [100a6607] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 22f1388e >2012-09-02 15:25:47,476 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [100a6607] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:25:47,477 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-87) [100a6607] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 22f1388e >2012-09-02 15:25:51,312 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [5bec48ce] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:25:57,514 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:25:57,534 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-8) START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 6ee74ea8 >2012-09-02 15:25:58,494 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-8) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:25:58,494 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-8) Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:25:58,494 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-8) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:25:58,494 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-8) FINISH, ConnectStoragePoolVDSCommand, log id: 6ee74ea8 >2012-09-02 15:25:58,494 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:25:58,495 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:25:58,566 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-8) [51fa4809] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:25:58,602 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-8) [51fa4809] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6b86592d >2012-09-02 15:25:58,602 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [51fa4809] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:25:58,602 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-8) [51fa4809] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6b86592d >2012-09-02 15:26:01,620 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-9) domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:26:08,644 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:26:08,661 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-23) START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 203748af >2012-09-02 15:26:09,586 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-23) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:26:09,586 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-23) Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:26:09,586 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-23) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:26:09,586 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-23) FINISH, ConnectStoragePoolVDSCommand, log id: 203748af >2012-09-02 15:26:09,586 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:26:09,586 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:26:10,888 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-23) [426fd482] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:26:10,944 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-23) [426fd482] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3df12c60 >2012-09-02 15:26:10,944 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [426fd482] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:26:10,944 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-23) [426fd482] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3df12c60 >2012-09-02 15:26:12,232 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-20) domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:26:21,005 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:26:21,023 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-39) START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 6f9051c9 >2012-09-02 15:26:21,865 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-39) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:26:21,865 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-39) Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:26:21,865 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-39) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:26:21,865 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-39) FINISH, ConnectStoragePoolVDSCommand, log id: 6f9051c9 >2012-09-02 15:26:21,865 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:26:21,865 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:26:21,985 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-39) [4e3557b5] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:26:22,015 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-39) [4e3557b5] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 32743420 >2012-09-02 15:26:22,015 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [4e3557b5] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:26:22,015 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-39) [4e3557b5] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 32743420 >2012-09-02 15:26:22,434 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-37) [36253582] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:26:32,052 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:26:32,063 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-51) START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 782a7868 >2012-09-02 15:26:32,892 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-51) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:26:32,892 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-51) Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:26:32,893 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-51) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:26:32,893 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-51) FINISH, ConnectStoragePoolVDSCommand, log id: 782a7868 >2012-09-02 15:26:32,893 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:26:32,893 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:26:33,309 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-51) [5b1ea8bd] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:26:33,345 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-51) [5b1ea8bd] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7b8cb828 >2012-09-02 15:26:33,345 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) [5b1ea8bd] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:26:33,345 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-51) [5b1ea8bd] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7b8cb828 >2012-09-02 15:26:42,951 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:26:43,494 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-64) hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:26:43,527 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-64) START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 2b0313f1 >2012-09-02 15:26:44,357 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-64) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:26:44,357 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-64) Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:26:44,357 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-64) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:26:44,357 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-64) FINISH, ConnectStoragePoolVDSCommand, log id: 2b0313f1 >2012-09-02 15:26:44,357 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-64) IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:26:44,357 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-64) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:26:45,347 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-64) [73b5feec] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:26:45,382 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-64) [73b5feec] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 32c10ced >2012-09-02 15:26:45,382 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-64) [73b5feec] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:26:45,382 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-64) [73b5feec] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 32c10ced >2012-09-02 15:26:53,207 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:26:55,420 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:26:55,437 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-90) START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 3835347f >2012-09-02 15:26:56,243 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-90) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:26:56,243 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-90) Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:26:56,243 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-90) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:26:56,243 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-90) FINISH, ConnectStoragePoolVDSCommand, log id: 3835347f >2012-09-02 15:26:56,244 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:26:56,244 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:26:56,319 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-90) [22e212c1] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:26:56,352 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-90) [22e212c1] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2ffa4fe7 >2012-09-02 15:26:56,352 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [22e212c1] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:26:56,352 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-90) [22e212c1] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2ffa4fe7 >2012-09-02 15:27:03,855 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [536973ff] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:27:06,390 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:27:06,522 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-1) START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 27c8493 >2012-09-02 15:27:07,353 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-1) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:27:07,353 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-1) Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:27:07,353 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-1) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:27:07,353 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-1) FINISH, ConnectStoragePoolVDSCommand, log id: 27c8493 >2012-09-02 15:27:07,353 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:27:07,353 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:27:07,519 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-1) [26e2c302] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:27:07,580 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-1) [26e2c302] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1a3c1541 >2012-09-02 15:27:07,580 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [26e2c302] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:27:07,580 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-1) [26e2c302] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1a3c1541 >2012-09-02 15:27:14,518 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-11) domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:27:17,624 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:27:17,696 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-14) START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 2cd0c64 >2012-09-02 15:27:18,514 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-14) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:27:18,514 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-14) Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:27:18,514 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-14) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:27:18,514 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-14) FINISH, ConnectStoragePoolVDSCommand, log id: 2cd0c64 >2012-09-02 15:27:18,515 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:27:18,515 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:27:18,618 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-14) [2ff7a817] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:27:18,648 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-14) [2ff7a817] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 120081e6 >2012-09-02 15:27:18,649 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [2ff7a817] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:27:18,649 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-14) [2ff7a817] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 120081e6 >2012-09-02 15:27:25,158 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-29) domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:27:28,696 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) [3fd0f791] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:27:28,712 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-31) [3fd0f791] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: cd32da6 >2012-09-02 15:27:29,532 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-31) [3fd0f791] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:27:29,532 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-31) [3fd0f791] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:27:29,532 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-31) [3fd0f791] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:27:29,532 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-31) [3fd0f791] FINISH, ConnectStoragePoolVDSCommand, log id: cd32da6 >2012-09-02 15:27:29,533 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) [3fd0f791] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:27:29,533 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) [3fd0f791] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:27:29,637 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-31) [227b1500] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:27:29,815 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-31) [227b1500] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: bfab80f >2012-09-02 15:27:29,815 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) [227b1500] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:27:29,815 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-31) [227b1500] FINISH, MarkPoolInReconstructModeVDSCommand, log id: bfab80f >2012-09-02 15:27:35,405 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [48816517] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:27:39,904 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:27:39,952 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-48) START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 4c1fc6de >2012-09-02 15:27:40,790 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-48) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:27:40,790 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-48) Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:27:40,790 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-48) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:27:40,790 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-48) FINISH, ConnectStoragePoolVDSCommand, log id: 4c1fc6de >2012-09-02 15:27:40,790 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:27:40,790 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:27:41,066 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-48) [7509bad3] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:27:41,135 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-48) [7509bad3] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7bb75e1 >2012-09-02 15:27:41,135 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [7509bad3] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:27:41,135 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-48) [7509bad3] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7bb75e1 >2012-09-02 15:27:45,768 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) [594f6f51] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:27:51,170 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [1138eff2] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:27:51,194 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-59) [1138eff2] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 64bc799b >2012-09-02 15:27:52,018 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-59) [1138eff2] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:27:52,018 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-59) [1138eff2] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:27:52,018 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-59) [1138eff2] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:27:52,018 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-59) [1138eff2] FINISH, ConnectStoragePoolVDSCommand, log id: 64bc799b >2012-09-02 15:27:52,018 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [1138eff2] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:27:52,018 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [1138eff2] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:27:52,120 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-59) [73165b38] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:27:52,187 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-59) [73165b38] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 55640f7d >2012-09-02 15:27:52,187 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [73165b38] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:27:52,187 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-59) [73165b38] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 55640f7d >2012-09-02 15:27:56,135 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-79) domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:28:02,250 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-76) hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:28:02,270 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-76) START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 6f2a2917 >2012-09-02 15:28:03,109 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-76) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:28:03,109 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-76) Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:28:03,109 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-76) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:28:03,109 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-76) FINISH, ConnectStoragePoolVDSCommand, log id: 6f2a2917 >2012-09-02 15:28:03,109 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-76) IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:28:03,109 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-76) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:28:03,283 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-76) [a637d5a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:28:03,344 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-76) [a637d5a] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2a5b3755 >2012-09-02 15:28:03,344 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-76) [a637d5a] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:28:03,344 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-76) [a637d5a] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2a5b3755 >2012-09-02 15:28:06,471 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:28:13,385 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [5bec48ce] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:28:13,401 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-98) [5bec48ce] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: b9dd924 >2012-09-02 15:28:14,243 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-98) [5bec48ce] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:28:14,243 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-98) [5bec48ce] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:28:14,243 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-98) [5bec48ce] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:28:14,243 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-98) [5bec48ce] FINISH, ConnectStoragePoolVDSCommand, log id: b9dd924 >2012-09-02 15:28:14,243 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [5bec48ce] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:28:14,243 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [5bec48ce] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:28:14,671 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-98) [7952176] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:28:15,083 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-98) [7952176] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 33c5ca27 >2012-09-02 15:28:15,083 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [7952176] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:28:15,084 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-98) [7952176] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 33c5ca27 >2012-09-02 15:28:16,733 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:28:25,177 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-11) hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:28:25,274 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-11) START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 435a3e98 >2012-09-02 15:28:26,087 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-11) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:28:26,087 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-11) Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:28:26,087 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-11) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:28:26,087 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-11) FINISH, ConnectStoragePoolVDSCommand, log id: 435a3e98 >2012-09-02 15:28:26,087 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-11) IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:28:26,087 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-11) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:28:26,200 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-11) [5246647c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:28:26,233 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-11) [5246647c] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 30fc573e >2012-09-02 15:28:26,233 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-11) [5246647c] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:28:26,233 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-11) [5246647c] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 30fc573e >2012-09-02 15:28:26,939 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-18) domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:28:36,277 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-28) hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:28:36,301 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-28) START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 22817ad0 >2012-09-02 15:28:36,903 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) starting ProcessDomainRecovery for domain 56a64ce1-90ab-4576-86fa-c269160aaa38 >2012-09-02 15:28:36,939 WARN [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (QuartzScheduler_Worker-30) [1e6caf94] The message key HandleFailedStorageDomain is missing from bundles/ExecutionMessages >2012-09-02 15:28:36,998 INFO [org.ovirt.engine.core.bll.storage.HandleFailedStorageDomainCommand] (QuartzScheduler_Worker-30) [1e6caf94] Running command: HandleFailedStorageDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:28:37,012 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-30) [117f14a2] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:28:37,122 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-28) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:28:37,122 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-28) Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:28:37,122 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-28) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:28:37,122 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-28) FINISH, ConnectStoragePoolVDSCommand, log id: 22817ad0 >2012-09-02 15:28:37,122 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-28) IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:28:37,122 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-28) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:28:37,558 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (QuartzScheduler_Worker-30) [117f14a2] START, DisconnectStorageServerVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, storageType = ISCSI, connectionList = [{ id: 90cee3d3-f602-49f9-8f17-8665c22f2d06, connection: 10.35.64.10 };]), log id: 6fc85bff >2012-09-02 15:28:38,166 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (QuartzScheduler_Worker-30) [117f14a2] FINISH, DisconnectStorageServerVDSCommand, return: {90cee3d3-f602-49f9-8f17-8665c22f2d06=0}, log id: 6fc85bff >2012-09-02 15:28:38,166 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-30) [117f14a2] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5a034c99 >2012-09-02 15:28:38,166 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [117f14a2] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:28:38,166 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-30) [117f14a2] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5a034c99 >2012-09-02 15:28:38,177 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-30) [117f14a2] No string for UNASSIGNED type. Use default Log >2012-09-02 15:28:47,151 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-44) [7f514ddd] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:28:47,166 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-44) [7f514ddd] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 3d9f13d2 >2012-09-02 15:28:48,004 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-44) [7f514ddd] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:28:48,005 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-44) [7f514ddd] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:28:48,005 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-44) [7f514ddd] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:28:48,005 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-44) [7f514ddd] FINISH, ConnectStoragePoolVDSCommand, log id: 3d9f13d2 >2012-09-02 15:28:48,005 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-44) [7f514ddd] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:28:48,005 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-44) [7f514ddd] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:28:48,105 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-44) [d375719] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:28:48,148 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-44) [d375719] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 42af516e >2012-09-02 15:28:48,148 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-44) [d375719] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:28:48,148 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-44) [d375719] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 42af516e >2012-09-02 15:28:58,197 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [330d4585] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:28:58,241 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-57) [330d4585] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 21e9d2a2 >2012-09-02 15:28:59,085 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-57) [330d4585] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:28:59,085 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-57) [330d4585] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:28:59,085 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-57) [330d4585] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:28:59,085 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-57) [330d4585] FINISH, ConnectStoragePoolVDSCommand, log id: 21e9d2a2 >2012-09-02 15:28:59,085 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [330d4585] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:28:59,085 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [330d4585] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:28:59,272 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-57) [515da613] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:28:59,366 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-57) [515da613] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 17c8e4a >2012-09-02 15:28:59,366 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [515da613] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:28:59,366 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-57) [515da613] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 17c8e4a >2012-09-02 15:29:09,424 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [46fa81bd] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:29:09,442 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-77) [46fa81bd] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 55d97c5e >2012-09-02 15:29:10,274 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-77) [46fa81bd] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:29:10,274 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-77) [46fa81bd] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:29:10,274 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-77) [46fa81bd] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:29:10,274 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-77) [46fa81bd] FINISH, ConnectStoragePoolVDSCommand, log id: 55d97c5e >2012-09-02 15:29:10,275 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [46fa81bd] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:29:10,275 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [46fa81bd] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:29:10,902 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-77) [50696184] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:29:10,946 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-77) [50696184] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 22cc7ec2 >2012-09-02 15:29:10,946 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [50696184] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:29:10,946 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-77) [50696184] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 22cc7ec2 >2012-09-02 15:29:21,094 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:29:21,198 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-97) START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 4f6c355c >2012-09-02 15:29:22,050 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-97) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:29:22,050 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-97) Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:29:22,050 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-97) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:29:22,050 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-97) FINISH, ConnectStoragePoolVDSCommand, log id: 4f6c355c >2012-09-02 15:29:22,051 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:29:22,051 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:29:22,148 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-97) [7473feec] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:29:22,189 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-97) [7473feec] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6fb4c9a >2012-09-02 15:29:22,189 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [7473feec] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:29:22,189 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-97) [7473feec] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6fb4c9a >2012-09-02 15:29:32,250 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-15) hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:29:32,265 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-15) START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 1502aa86 >2012-09-02 15:29:33,103 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-15) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:29:33,103 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-15) Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:29:33,103 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-15) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:29:33,103 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-15) FINISH, ConnectStoragePoolVDSCommand, log id: 1502aa86 >2012-09-02 15:29:33,103 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-15) IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:29:33,103 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-15) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:29:33,170 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-15) [10bf0b7b] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:29:33,204 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-15) [10bf0b7b] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 208c9bfd >2012-09-02 15:29:33,204 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-15) [10bf0b7b] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:29:33,204 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-15) [10bf0b7b] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 208c9bfd >2012-09-02 15:29:43,246 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-28) hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:29:43,263 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-28) START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 43f0c91d >2012-09-02 15:29:44,094 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-28) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:29:44,094 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-28) Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:29:44,094 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-28) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:29:44,094 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-28) FINISH, ConnectStoragePoolVDSCommand, log id: 43f0c91d >2012-09-02 15:29:44,094 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-28) IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:29:44,094 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-28) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:29:44,205 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-28) [45c77945] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:29:44,242 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-28) [45c77945] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 406b6ad7 >2012-09-02 15:29:44,242 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-28) [45c77945] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:29:44,242 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-28) [45c77945] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 406b6ad7 >2012-09-02 15:29:54,288 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:29:54,302 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-50) START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 58001696 >2012-09-02 15:29:55,159 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:29:55,160 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-50) Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:29:55,160 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:29:55,160 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-50) FINISH, ConnectStoragePoolVDSCommand, log id: 58001696 >2012-09-02 15:29:55,160 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:29:55,160 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:29:55,243 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-50) [61b8f16c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:29:55,282 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-50) [61b8f16c] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1aef75ca >2012-09-02 15:29:55,282 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [61b8f16c] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:29:55,282 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-50) [61b8f16c] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1aef75ca >2012-09-02 15:30:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-60) Autorecovering hosts is disabled, skipping >2012-09-02 15:30:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-60) Autorecovering storage domains is disabled, skipping >2012-09-02 15:30:05,326 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:30:05,354 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-63) START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 1520f147 >2012-09-02 15:30:06,243 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-63) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:30:06,243 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-63) Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:30:06,243 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-63) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:30:06,243 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-63) FINISH, ConnectStoragePoolVDSCommand, log id: 1520f147 >2012-09-02 15:30:06,243 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:30:06,244 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:30:06,987 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-63) [6267f606] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:30:07,027 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-63) [6267f606] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4f752391 >2012-09-02 15:30:07,027 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) [6267f606] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:30:07,027 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-63) [6267f606] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4f752391 >2012-09-02 15:30:17,072 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:30:17,132 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-86) START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 2144911 >2012-09-02 15:30:17,977 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-86) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:30:17,977 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-86) Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:30:17,977 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-86) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:30:17,977 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-86) FINISH, ConnectStoragePoolVDSCommand, log id: 2144911 >2012-09-02 15:30:17,977 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:30:17,977 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:30:18,156 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-86) [11e497be] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:30:18,210 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-86) [11e497be] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 297f6597 >2012-09-02 15:30:18,210 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) [11e497be] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:30:18,210 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-86) [11e497be] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 297f6597 >2012-09-02 15:30:28,269 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [7473feec] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:30:28,285 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-97) [7473feec] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 5e974a30 >2012-09-02 15:30:29,133 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-97) [7473feec] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:30:29,133 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-97) [7473feec] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:30:29,133 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-97) [7473feec] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:30:29,134 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-97) [7473feec] FINISH, ConnectStoragePoolVDSCommand, log id: 5e974a30 >2012-09-02 15:30:29,134 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [7473feec] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:30:29,134 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [7473feec] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:30:29,230 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-97) [793b3360] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:30:29,260 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-97) [793b3360] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 144d96f9 >2012-09-02 15:30:29,260 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [793b3360] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:30:29,260 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-97) [793b3360] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 144d96f9 >2012-09-02 15:30:39,300 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-11) [5246647c] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:30:39,316 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-11) [5246647c] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 7abe677 >2012-09-02 15:30:40,156 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-11) [5246647c] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:30:40,156 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-11) [5246647c] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:30:40,157 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-11) [5246647c] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:30:40,157 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-11) [5246647c] FINISH, ConnectStoragePoolVDSCommand, log id: 7abe677 >2012-09-02 15:30:40,157 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-11) [5246647c] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:30:40,157 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-11) [5246647c] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:30:40,244 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-11) [13caede9] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:30:40,283 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-11) [13caede9] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: a5e9b74 >2012-09-02 15:30:40,283 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-11) [13caede9] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:30:40,283 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-11) [13caede9] FINISH, MarkPoolInReconstructModeVDSCommand, log id: a5e9b74 >2012-09-02 15:30:50,322 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [117f14a2] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:30:50,336 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-30) [117f14a2] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 72f3a9f7 >2012-09-02 15:30:51,155 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-30) [117f14a2] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:30:51,155 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-30) [117f14a2] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:30:51,155 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-30) [117f14a2] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:30:51,155 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-30) [117f14a2] FINISH, ConnectStoragePoolVDSCommand, log id: 72f3a9f7 >2012-09-02 15:30:51,155 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [117f14a2] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:30:51,155 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [117f14a2] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:30:51,230 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-30) [4722d115] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:30:51,284 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-30) [4722d115] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4fbc9748 >2012-09-02 15:30:51,284 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [4722d115] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:30:51,284 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-30) [4722d115] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4fbc9748 >2012-09-02 15:31:01,342 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [61b8f16c] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:31:01,362 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-50) [61b8f16c] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 9fa2150 >2012-09-02 15:31:02,188 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-50) [61b8f16c] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:31:02,189 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-50) [61b8f16c] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:31:02,189 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-50) [61b8f16c] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:31:02,189 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-50) [61b8f16c] FINISH, ConnectStoragePoolVDSCommand, log id: 9fa2150 >2012-09-02 15:31:02,189 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [61b8f16c] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:31:02,189 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [61b8f16c] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:31:02,277 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-50) [308ea3f7] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:31:02,341 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-50) [308ea3f7] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 14e1d12a >2012-09-02 15:31:02,341 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [308ea3f7] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:31:02,341 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-50) [308ea3f7] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 14e1d12a >2012-09-02 15:31:12,394 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [515da613] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:31:12,413 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-57) [515da613] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 2cc2093f >2012-09-02 15:31:13,245 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-57) [515da613] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:31:13,245 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-57) [515da613] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:31:13,245 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-57) [515da613] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:31:13,245 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-57) [515da613] FINISH, ConnectStoragePoolVDSCommand, log id: 2cc2093f >2012-09-02 15:31:13,245 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [515da613] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:31:13,245 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [515da613] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:31:13,341 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-57) [525b5903] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:31:13,391 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-57) [525b5903] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 42515828 >2012-09-02 15:31:13,391 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [525b5903] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:31:13,391 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-57) [525b5903] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 42515828 >2012-09-02 15:31:23,443 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [50696184] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:31:23,498 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-77) [50696184] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 4ff5f91a >2012-09-02 15:31:24,333 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-77) [50696184] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:31:24,333 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-77) [50696184] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:31:24,334 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-77) [50696184] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:31:24,334 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-77) [50696184] FINISH, ConnectStoragePoolVDSCommand, log id: 4ff5f91a >2012-09-02 15:31:24,334 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [50696184] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:31:24,334 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [50696184] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:31:24,407 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-77) [6423388b] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:31:24,443 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-77) [6423388b] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 47140715 >2012-09-02 15:31:24,443 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [6423388b] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:31:24,443 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-77) [6423388b] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 47140715 >2012-09-02 15:31:34,519 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-78) hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:31:34,535 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-78) START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 381256ce >2012-09-02 15:31:35,373 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-78) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:31:35,761 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-78) Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:31:35,761 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-78) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:31:35,761 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-78) FINISH, ConnectStoragePoolVDSCommand, log id: 381256ce >2012-09-02 15:31:35,762 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-78) IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:31:35,762 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-78) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:31:37,323 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-78) [6a1d44c4] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:31:37,406 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-78) [6a1d44c4] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4f7fa799 >2012-09-02 15:31:37,406 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-78) [6a1d44c4] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:31:37,406 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-78) [6a1d44c4] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4f7fa799 >2012-09-02 15:31:47,473 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:31:47,506 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-21) START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 6169a7b >2012-09-02 15:31:48,338 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-21) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:31:48,338 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-21) Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:31:48,338 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-21) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:31:48,338 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-21) FINISH, ConnectStoragePoolVDSCommand, log id: 6169a7b >2012-09-02 15:31:48,339 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:31:48,339 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:31:48,915 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-21) [6efb5251] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:31:48,952 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-21) [6efb5251] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7d96fe88 >2012-09-02 15:31:48,952 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) [6efb5251] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:31:48,952 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-21) [6efb5251] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7d96fe88 >2012-09-02 15:31:58,999 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-32) hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:31:59,009 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-32) START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 68a93ee4 >2012-09-02 15:31:59,840 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-32) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:31:59,840 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-32) Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:31:59,840 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-32) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:31:59,840 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-32) FINISH, ConnectStoragePoolVDSCommand, log id: 68a93ee4 >2012-09-02 15:31:59,840 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-32) IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:31:59,840 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-32) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:32:00,051 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-32) [65b08d21] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:32:00,088 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-32) [65b08d21] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5c6b901c >2012-09-02 15:32:00,088 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-32) [65b08d21] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:32:00,088 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-32) [65b08d21] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5c6b901c >2012-09-02 15:32:10,146 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-52) hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:32:10,363 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-52) START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 3d44ef03 >2012-09-02 15:32:11,214 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-52) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:32:11,214 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-52) Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:32:11,214 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-52) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:32:11,214 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-52) FINISH, ConnectStoragePoolVDSCommand, log id: 3d44ef03 >2012-09-02 15:32:11,214 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-52) IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:32:11,214 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-52) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:32:11,290 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-52) [1d401342] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:32:11,330 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-52) [1d401342] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7340e759 >2012-09-02 15:32:11,330 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-52) [1d401342] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:32:11,330 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-52) [1d401342] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7340e759 >2012-09-02 15:32:21,374 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-67) [fb71dbe] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:32:21,556 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-67) [fb71dbe] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 20d09f1e >2012-09-02 15:32:22,388 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-67) [fb71dbe] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:32:22,388 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-67) [fb71dbe] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:32:22,388 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-67) [fb71dbe] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:32:22,388 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-67) [fb71dbe] FINISH, ConnectStoragePoolVDSCommand, log id: 20d09f1e >2012-09-02 15:32:22,388 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-67) [fb71dbe] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:32:22,388 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-67) [fb71dbe] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:32:22,468 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-67) [25620c1f] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:32:22,527 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-67) [25620c1f] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 17501061 >2012-09-02 15:32:22,527 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-67) [25620c1f] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:32:22,527 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-67) [25620c1f] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 17501061 >2012-09-02 15:32:32,583 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [28a39dff] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:32:32,598 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-83) [28a39dff] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 730ecb69 >2012-09-02 15:32:33,420 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-83) [28a39dff] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:32:33,420 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-83) [28a39dff] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:32:33,420 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-83) [28a39dff] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:32:33,420 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-83) [28a39dff] FINISH, ConnectStoragePoolVDSCommand, log id: 730ecb69 >2012-09-02 15:32:33,420 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [28a39dff] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:32:33,420 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [28a39dff] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:32:34,181 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-83) [21dfea96] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:32:34,284 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-83) [21dfea96] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 11616204 >2012-09-02 15:32:34,284 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [21dfea96] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:32:34,284 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-83) [21dfea96] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 11616204 >2012-09-02 15:32:44,371 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-100) hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:32:44,445 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-100) START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 3ce9bbcf >2012-09-02 15:32:45,273 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-100) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:32:45,273 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-100) Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:32:45,273 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-100) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:32:45,273 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-100) FINISH, ConnectStoragePoolVDSCommand, log id: 3ce9bbcf >2012-09-02 15:32:45,273 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-100) IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:32:45,273 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-100) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:32:45,378 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-100) [5bcc0093] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:32:45,412 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-100) [5bcc0093] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4481f85d >2012-09-02 15:32:45,412 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-100) [5bcc0093] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:32:45,412 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-100) [5bcc0093] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4481f85d >2012-09-02 15:32:55,457 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-19) hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:32:55,488 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-19) START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 21fa3071 >2012-09-02 15:32:56,309 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-19) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:32:56,309 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-19) Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:32:56,309 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-19) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:32:56,309 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-19) FINISH, ConnectStoragePoolVDSCommand, log id: 21fa3071 >2012-09-02 15:32:56,309 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-19) IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:32:56,309 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-19) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:32:56,443 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-19) [29d285ee] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:32:56,497 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-19) [29d285ee] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5a9280f9 >2012-09-02 15:32:56,497 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-19) [29d285ee] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:32:56,497 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-19) [29d285ee] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5a9280f9 >2012-09-02 15:33:06,547 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [4e3557b5] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:33:06,923 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-39) [4e3557b5] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: cb4065d >2012-09-02 15:33:07,762 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-39) [4e3557b5] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:33:07,762 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-39) [4e3557b5] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:33:07,762 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-39) [4e3557b5] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:33:07,763 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-39) [4e3557b5] FINISH, ConnectStoragePoolVDSCommand, log id: cb4065d >2012-09-02 15:33:07,763 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [4e3557b5] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:33:07,763 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [4e3557b5] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:33:07,871 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-39) [aa7db89] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:33:07,911 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-39) [aa7db89] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 52fe7f95 >2012-09-02 15:33:07,911 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [aa7db89] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:33:07,911 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-39) [aa7db89] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 52fe7f95 >2012-09-02 15:33:17,950 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) [5b1ea8bd] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:33:18,135 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-51) [5b1ea8bd] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 2c8af408 >2012-09-02 15:33:18,981 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-51) [5b1ea8bd] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:33:18,981 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-51) [5b1ea8bd] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:33:18,981 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-51) [5b1ea8bd] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:33:18,981 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-51) [5b1ea8bd] FINISH, ConnectStoragePoolVDSCommand, log id: 2c8af408 >2012-09-02 15:33:18,981 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) [5b1ea8bd] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:33:18,981 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) [5b1ea8bd] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:33:19,529 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-51) [5df6ca7d] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:33:19,574 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-51) [5df6ca7d] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 731886d6 >2012-09-02 15:33:19,575 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) [5df6ca7d] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:33:19,575 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-51) [5df6ca7d] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 731886d6 >2012-09-02 15:33:29,613 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-75) hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:33:29,631 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-75) START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 668040de >2012-09-02 15:33:30,479 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-75) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:33:30,479 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-75) Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:33:30,479 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-75) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:33:30,479 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-75) FINISH, ConnectStoragePoolVDSCommand, log id: 668040de >2012-09-02 15:33:30,479 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-75) IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:33:30,479 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-75) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:33:30,564 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-75) [2ec70fed] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:33:30,596 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-75) [2ec70fed] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 54acba21 >2012-09-02 15:33:30,596 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-75) [2ec70fed] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:33:30,596 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-75) [2ec70fed] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 54acba21 >2012-09-02 15:33:40,646 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [21dfea96] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:33:41,010 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-83) [21dfea96] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: da033fa >2012-09-02 15:33:42,064 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-83) [21dfea96] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:33:42,064 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-83) [21dfea96] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:33:42,064 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-83) [21dfea96] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:33:42,064 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-83) [21dfea96] FINISH, ConnectStoragePoolVDSCommand, log id: da033fa >2012-09-02 15:33:42,064 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [21dfea96] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:33:42,064 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [21dfea96] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:33:42,416 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-83) [2863ec16] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:33:42,451 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-83) [2863ec16] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4a2b61bd >2012-09-02 15:33:42,452 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [2863ec16] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:33:42,452 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-83) [2863ec16] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4a2b61bd >2012-09-02 15:33:52,496 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [51fa4809] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:33:52,523 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-8) [51fa4809] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 412197a0 >2012-09-02 15:33:53,460 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-8) [51fa4809] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:33:53,460 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-8) [51fa4809] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:33:53,460 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-8) [51fa4809] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:33:53,460 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-8) [51fa4809] FINISH, ConnectStoragePoolVDSCommand, log id: 412197a0 >2012-09-02 15:33:53,460 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [51fa4809] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:33:53,460 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [51fa4809] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:33:53,598 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-8) [97db800] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:33:53,646 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-8) [97db800] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 903241 >2012-09-02 15:33:53,646 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [97db800] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:33:53,646 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-8) [97db800] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 903241 >2012-09-02 15:34:03,713 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [426fd482] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:34:03,928 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-23) [426fd482] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 283b465b >2012-09-02 15:34:04,772 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-23) [426fd482] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:34:04,772 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-23) [426fd482] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:34:04,772 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-23) [426fd482] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:34:04,772 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-23) [426fd482] FINISH, ConnectStoragePoolVDSCommand, log id: 283b465b >2012-09-02 15:34:04,772 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [426fd482] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:34:04,772 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [426fd482] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:34:05,422 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-23) [6b139e0b] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:34:05,484 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-23) [6b139e0b] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4ca03fbf >2012-09-02 15:34:05,484 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [6b139e0b] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:34:05,484 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-23) [6b139e0b] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4ca03fbf >2012-09-02 15:34:15,527 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-37) [36253582] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:34:15,552 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-37) [36253582] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 27749c64 >2012-09-02 15:34:16,378 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-37) [36253582] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:34:16,378 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-37) [36253582] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:34:16,378 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-37) [36253582] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:34:16,378 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-37) [36253582] FINISH, ConnectStoragePoolVDSCommand, log id: 27749c64 >2012-09-02 15:34:16,378 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-37) [36253582] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:34:16,378 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-37) [36253582] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:34:16,463 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-37) [14f6fa8e] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:34:16,495 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-37) [14f6fa8e] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2e1e075e >2012-09-02 15:34:16,495 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-37) [14f6fa8e] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:34:16,495 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-37) [14f6fa8e] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2e1e075e >2012-09-02 15:34:26,551 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:34:26,588 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-60) START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 5325dded >2012-09-02 15:34:27,419 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-60) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:34:27,420 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-60) Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:34:27,420 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-60) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:34:27,420 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-60) FINISH, ConnectStoragePoolVDSCommand, log id: 5325dded >2012-09-02 15:34:27,420 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:34:27,420 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:34:27,480 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-60) [6e66b462] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:34:27,540 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-60) [6e66b462] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 89d6ccf >2012-09-02 15:34:27,540 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) [6e66b462] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:34:27,540 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-60) [6e66b462] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 89d6ccf >2012-09-02 15:34:37,608 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:34:37,626 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-71) START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 6550eb92 >2012-09-02 15:34:38,469 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-71) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:34:38,469 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-71) Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:34:38,470 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-71) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:34:38,470 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-71) FINISH, ConnectStoragePoolVDSCommand, log id: 6550eb92 >2012-09-02 15:34:38,470 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:34:38,470 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:34:38,634 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-71) [698ff074] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:34:38,676 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-71) [698ff074] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 12094f7e >2012-09-02 15:34:38,676 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [698ff074] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:34:38,676 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-71) [698ff074] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 12094f7e >2012-09-02 15:34:48,719 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:34:48,747 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-85) START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 616a9824 >2012-09-02 15:34:49,579 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-85) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:34:49,818 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-85) Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:34:49,818 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-85) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:34:49,818 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-85) FINISH, ConnectStoragePoolVDSCommand, log id: 616a9824 >2012-09-02 15:34:49,818 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:34:49,818 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:34:50,319 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-85) [5e430da1] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:34:50,400 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-85) [5e430da1] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7c542c19 >2012-09-02 15:34:50,400 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [5e430da1] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:34:50,400 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-85) [5e430da1] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7c542c19 >2012-09-02 15:35:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-2) Autorecovering hosts is disabled, skipping >2012-09-02 15:35:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-2) Autorecovering storage domains is disabled, skipping >2012-09-02 15:35:00,440 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [26e2c302] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:35:00,458 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-1) [26e2c302] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 48f56034 >2012-09-02 15:35:01,305 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-1) [26e2c302] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:35:01,305 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-1) [26e2c302] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:35:01,305 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-1) [26e2c302] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:35:01,305 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-1) [26e2c302] FINISH, ConnectStoragePoolVDSCommand, log id: 48f56034 >2012-09-02 15:35:01,305 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [26e2c302] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:35:01,305 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [26e2c302] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:35:01,473 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-1) [1f04073e] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:35:01,509 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-1) [1f04073e] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 155899d0 >2012-09-02 15:35:01,509 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [1f04073e] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:35:01,509 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-1) [1f04073e] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 155899d0 >2012-09-02 15:35:11,743 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:35:11,759 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-17) START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 52eeabc3 >2012-09-02 15:35:12,556 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-17) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:35:12,556 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-17) Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:35:12,556 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-17) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:35:12,556 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-17) FINISH, ConnectStoragePoolVDSCommand, log id: 52eeabc3 >2012-09-02 15:35:12,556 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:35:12,556 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:35:12,643 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-17) [4d5ce074] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:35:12,707 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-17) [4d5ce074] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 238db118 >2012-09-02 15:35:12,707 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [4d5ce074] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:35:12,707 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-17) [4d5ce074] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 238db118 >2012-09-02 15:35:22,747 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-35) [372f1ff9] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:35:22,763 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-35) [372f1ff9] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 307244a5 >2012-09-02 15:35:23,606 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-35) [372f1ff9] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:35:23,607 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-35) [372f1ff9] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:35:23,607 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-35) [372f1ff9] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:35:23,607 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-35) [372f1ff9] FINISH, ConnectStoragePoolVDSCommand, log id: 307244a5 >2012-09-02 15:35:23,607 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-35) [372f1ff9] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:35:23,607 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-35) [372f1ff9] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:35:23,691 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-35) [6a98ae20] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:35:23,718 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-35) [6a98ae20] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 52359f47 >2012-09-02 15:35:23,718 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-35) [6a98ae20] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:35:23,718 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-35) [6a98ae20] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 52359f47 >2012-09-02 15:35:33,758 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-44) [d375719] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:35:34,209 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-44) [d375719] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 130ff778 >2012-09-02 15:35:35,053 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-44) [d375719] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:35:35,053 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-44) [d375719] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:35:35,053 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-44) [d375719] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:35:35,053 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-44) [d375719] FINISH, ConnectStoragePoolVDSCommand, log id: 130ff778 >2012-09-02 15:35:35,053 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-44) [d375719] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:35:35,053 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-44) [d375719] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:35:35,163 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-44) [2ec8fb1f] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:35:35,217 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-44) [2ec8fb1f] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: c6c871a >2012-09-02 15:35:35,217 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-44) [2ec8fb1f] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:35:35,217 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-44) [2ec8fb1f] FINISH, MarkPoolInReconstructModeVDSCommand, log id: c6c871a >2012-09-02 15:35:45,266 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:35:45,808 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-66) START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 3f22caa1 >2012-09-02 15:35:46,666 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-66) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:35:46,666 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-66) Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:35:46,666 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-66) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:35:46,666 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-66) FINISH, ConnectStoragePoolVDSCommand, log id: 3f22caa1 >2012-09-02 15:35:46,666 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:35:46,666 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:35:46,808 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-66) [60860e0e] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:35:46,848 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-66) [60860e0e] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 678b169e >2012-09-02 15:35:46,849 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [60860e0e] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:35:46,849 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-66) [60860e0e] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 678b169e >2012-09-02 15:35:56,887 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [6423388b] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:35:56,904 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-77) [6423388b] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 76b6bdb6 >2012-09-02 15:35:57,765 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-77) [6423388b] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:35:57,765 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-77) [6423388b] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:35:57,765 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-77) [6423388b] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:35:57,765 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-77) [6423388b] FINISH, ConnectStoragePoolVDSCommand, log id: 76b6bdb6 >2012-09-02 15:35:57,766 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [6423388b] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:35:57,766 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [6423388b] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:35:57,871 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-77) [df2e2a7] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:35:57,932 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-77) [df2e2a7] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 797ab432 >2012-09-02 15:35:57,933 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [df2e2a7] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:35:57,933 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-77) [df2e2a7] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 797ab432 >2012-09-02 15:36:07,987 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [793b3360] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:36:08,279 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-97) [793b3360] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 2b0d4694 >2012-09-02 15:36:09,129 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-97) [793b3360] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:36:09,129 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-97) [793b3360] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:36:09,129 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-97) [793b3360] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:36:09,129 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-97) [793b3360] FINISH, ConnectStoragePoolVDSCommand, log id: 2b0d4694 >2012-09-02 15:36:09,129 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [793b3360] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:36:09,129 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [793b3360] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:36:09,287 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-97) [5ed2c82c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:36:09,320 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-97) [5ed2c82c] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 44bf8958 >2012-09-02 15:36:09,320 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [5ed2c82c] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:36:09,320 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-97) [5ed2c82c] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 44bf8958 >2012-09-02 15:36:19,360 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-15) [10bf0b7b] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:36:19,445 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-15) [10bf0b7b] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 1c2be611 >2012-09-02 15:36:20,282 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-15) [10bf0b7b] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:36:20,282 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-15) [10bf0b7b] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:36:20,282 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-15) [10bf0b7b] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:36:20,282 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-15) [10bf0b7b] FINISH, ConnectStoragePoolVDSCommand, log id: 1c2be611 >2012-09-02 15:36:20,282 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-15) [10bf0b7b] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:36:20,282 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-15) [10bf0b7b] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:36:20,406 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-15) [d40d6cd] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:36:20,452 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-15) [d40d6cd] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 36babb4c >2012-09-02 15:36:20,452 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-15) [d40d6cd] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:36:20,452 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-15) [d40d6cd] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 36babb4c >2012-09-02 15:36:30,522 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-40) hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:36:30,537 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-40) START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 221f4ed5 >2012-09-02 15:36:31,374 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-40) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:36:31,374 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-40) Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:36:31,374 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-40) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:36:31,374 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-40) FINISH, ConnectStoragePoolVDSCommand, log id: 221f4ed5 >2012-09-02 15:36:31,374 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-40) IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:36:31,374 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-40) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:36:31,462 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-40) [31e89cb7] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:36:31,493 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-40) [31e89cb7] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2d12a1c2 >2012-09-02 15:36:31,493 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-40) [31e89cb7] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:36:31,494 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-40) [31e89cb7] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2d12a1c2 >2012-09-02 15:36:41,543 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-45) hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:36:41,931 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-45) START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 6faf1b08 >2012-09-02 15:36:42,772 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-45) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:36:42,773 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-45) Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:36:42,773 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-45) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:36:42,773 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-45) FINISH, ConnectStoragePoolVDSCommand, log id: 6faf1b08 >2012-09-02 15:36:42,773 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-45) IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:36:42,773 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-45) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:36:42,910 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-45) [7b580d0f] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:36:42,969 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-45) [7b580d0f] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 245e3e5a >2012-09-02 15:36:42,969 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-45) [7b580d0f] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:36:42,969 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-45) [7b580d0f] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 245e3e5a >2012-09-02 15:36:53,018 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) [594f6f51] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:36:53,035 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-55) [594f6f51] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 4267e80e >2012-09-02 15:36:53,869 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-55) [594f6f51] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:36:53,869 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-55) [594f6f51] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:36:53,869 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-55) [594f6f51] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:36:53,869 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-55) [594f6f51] FINISH, ConnectStoragePoolVDSCommand, log id: 4267e80e >2012-09-02 15:36:53,869 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) [594f6f51] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:36:53,870 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) [594f6f51] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:36:53,950 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-55) [3942714e] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:36:53,978 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-55) [3942714e] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3a7f6ad5 >2012-09-02 15:36:53,978 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) [3942714e] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:36:53,978 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-55) [3942714e] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3a7f6ad5 >2012-09-02 15:37:04,089 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-80) hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:37:04,109 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-80) START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 46778f9 >2012-09-02 15:37:04,964 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-80) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:37:04,964 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-80) Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:37:04,964 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-80) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:37:04,964 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-80) FINISH, ConnectStoragePoolVDSCommand, log id: 46778f9 >2012-09-02 15:37:04,964 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-80) IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:37:04,964 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-80) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:37:05,367 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-80) [3692be71] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:37:05,432 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-80) [3692be71] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 262fdd67 >2012-09-02 15:37:05,432 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-80) [3692be71] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:37:05,432 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-80) [3692be71] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 262fdd67 >2012-09-02 15:37:15,477 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-95) [651f330f] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:37:15,492 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-95) [651f330f] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: c2c8c8d >2012-09-02 15:37:16,345 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-95) [651f330f] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:37:16,346 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-95) [651f330f] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:37:16,346 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-95) [651f330f] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:37:16,346 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-95) [651f330f] FINISH, ConnectStoragePoolVDSCommand, log id: c2c8c8d >2012-09-02 15:37:16,346 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-95) [651f330f] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:37:16,346 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-95) [651f330f] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:37:16,644 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-95) [4fcef51] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:37:16,694 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-95) [4fcef51] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: a1bf22b >2012-09-02 15:37:16,694 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-95) [4fcef51] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:37:16,694 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-95) [4fcef51] FINISH, MarkPoolInReconstructModeVDSCommand, log id: a1bf22b >2012-09-02 15:37:26,742 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-9) hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:37:26,757 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-9) START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 79a3cfe1 >2012-09-02 15:37:27,602 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-9) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:37:27,602 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-9) Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:37:27,602 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-9) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:37:27,602 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-9) FINISH, ConnectStoragePoolVDSCommand, log id: 79a3cfe1 >2012-09-02 15:37:27,602 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-9) IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:37:27,602 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-9) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:37:27,694 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-9) [6bd9f19d] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:37:27,732 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-9) [6bd9f19d] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 66a7836b >2012-09-02 15:37:27,732 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-9) [6bd9f19d] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:37:27,732 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-9) [6bd9f19d] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 66a7836b >2012-09-02 15:37:37,770 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [6b139e0b] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:37:37,796 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-23) [6b139e0b] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 66974509 >2012-09-02 15:37:38,654 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-23) [6b139e0b] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:37:38,654 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-23) [6b139e0b] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:37:38,654 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-23) [6b139e0b] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:37:38,654 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-23) [6b139e0b] FINISH, ConnectStoragePoolVDSCommand, log id: 66974509 >2012-09-02 15:37:38,654 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [6b139e0b] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:37:38,654 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [6b139e0b] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:37:38,728 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-23) [bedc93a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:37:38,763 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-23) [bedc93a] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1d258565 >2012-09-02 15:37:38,763 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [bedc93a] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:37:38,763 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-23) [bedc93a] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1d258565 >2012-09-02 15:37:48,803 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:37:49,332 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-38) START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: e90c0f2 >2012-09-02 15:37:50,174 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-38) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:37:50,174 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-38) Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:37:50,174 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-38) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:37:50,174 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-38) FINISH, ConnectStoragePoolVDSCommand, log id: e90c0f2 >2012-09-02 15:37:50,174 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:37:50,174 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:37:50,764 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-38) [195448f7] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:37:50,829 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-38) [195448f7] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5d66b78d >2012-09-02 15:37:50,829 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [195448f7] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:37:50,829 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-38) [195448f7] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5d66b78d >2012-09-02 15:38:00,886 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) [6e66b462] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:38:00,934 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-60) [6e66b462] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 403146c7 >2012-09-02 15:38:01,775 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-60) [6e66b462] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:38:01,775 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-60) [6e66b462] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:38:01,775 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-60) [6e66b462] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:38:01,775 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-60) [6e66b462] FINISH, ConnectStoragePoolVDSCommand, log id: 403146c7 >2012-09-02 15:38:01,775 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) [6e66b462] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:38:01,775 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) [6e66b462] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:38:01,852 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-60) [dbe32c9] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:38:01,895 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-60) [dbe32c9] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6282baba >2012-09-02 15:38:01,895 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) [dbe32c9] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:38:01,895 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-60) [dbe32c9] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6282baba >2012-09-02 15:38:11,934 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [698ff074] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:38:11,951 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-71) [698ff074] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 6251be50 >2012-09-02 15:38:12,781 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-71) [698ff074] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:38:12,781 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-71) [698ff074] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:38:12,781 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-71) [698ff074] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:38:12,781 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-71) [698ff074] FINISH, ConnectStoragePoolVDSCommand, log id: 6251be50 >2012-09-02 15:38:12,781 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [698ff074] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:38:12,781 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [698ff074] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:38:12,867 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-71) [53349194] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:38:12,905 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-71) [53349194] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 572b26a >2012-09-02 15:38:12,905 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [53349194] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:38:12,905 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-71) [53349194] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 572b26a >2012-09-02 15:38:22,942 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [22e212c1] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:38:22,964 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-90) [22e212c1] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 47d72f3c >2012-09-02 15:38:23,784 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-90) [22e212c1] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:38:23,784 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-90) [22e212c1] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:38:23,785 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-90) [22e212c1] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:38:23,785 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-90) [22e212c1] FINISH, ConnectStoragePoolVDSCommand, log id: 47d72f3c >2012-09-02 15:38:23,785 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [22e212c1] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:38:23,785 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [22e212c1] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:38:23,863 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-90) [5150576a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:38:23,899 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-90) [5150576a] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 69bb42e >2012-09-02 15:38:23,899 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [5150576a] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:38:23,899 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-90) [5150576a] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 69bb42e >2012-09-02 15:38:33,948 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:38:33,967 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-2) START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 3f64c989 >2012-09-02 15:38:34,796 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-2) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:38:34,796 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-2) Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:38:34,797 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-2) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:38:34,797 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-2) FINISH, ConnectStoragePoolVDSCommand, log id: 3f64c989 >2012-09-02 15:38:34,797 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:38:34,797 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:38:35,653 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-2) [45df5ae9] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:38:35,690 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-2) [45df5ae9] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 44c7bc7f >2012-09-02 15:38:35,690 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) [45df5ae9] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:38:35,690 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-2) [45df5ae9] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 44c7bc7f >2012-09-02 15:38:45,742 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) [6efb5251] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:38:45,920 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-21) [6efb5251] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 70b32d0 >2012-09-02 15:38:46,779 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-21) [6efb5251] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:38:46,779 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-21) [6efb5251] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:38:46,779 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-21) [6efb5251] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:38:46,779 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-21) [6efb5251] FINISH, ConnectStoragePoolVDSCommand, log id: 70b32d0 >2012-09-02 15:38:46,779 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) [6efb5251] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:38:46,779 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) [6efb5251] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:38:46,892 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-21) [ad5ebcb] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:38:46,937 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-21) [ad5ebcb] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2ee062cb >2012-09-02 15:38:46,937 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) [ad5ebcb] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:38:46,937 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-21) [ad5ebcb] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2ee062cb >2012-09-02 15:38:56,985 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:38:57,041 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-41) START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 14927cda >2012-09-02 15:38:57,888 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-41) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:38:57,888 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-41) Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:38:57,888 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-41) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:38:57,888 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-41) FINISH, ConnectStoragePoolVDSCommand, log id: 14927cda >2012-09-02 15:38:57,888 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:38:57,888 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:38:57,962 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-41) [2a140678] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:38:58,001 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-41) [2a140678] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4d654bed >2012-09-02 15:38:58,001 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) [2a140678] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:38:58,001 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-41) [2a140678] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4d654bed >2012-09-02 15:39:08,038 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-44) [2ec8fb1f] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:39:08,078 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-44) [2ec8fb1f] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 19ade7bb >2012-09-02 15:39:08,907 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-44) [2ec8fb1f] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:39:08,907 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-44) [2ec8fb1f] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:39:08,907 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-44) [2ec8fb1f] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:39:08,907 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-44) [2ec8fb1f] FINISH, ConnectStoragePoolVDSCommand, log id: 19ade7bb >2012-09-02 15:39:08,907 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-44) [2ec8fb1f] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:39:08,907 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-44) [2ec8fb1f] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:39:08,990 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-44) [30ec8757] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:39:09,022 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-44) [30ec8757] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2c53eda7 >2012-09-02 15:39:09,022 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-44) [30ec8757] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:39:09,022 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-44) [30ec8757] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2c53eda7 >2012-09-02 15:39:19,062 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [525b5903] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:39:19,077 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-57) [525b5903] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 4c852f2c >2012-09-02 15:39:19,948 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-57) [525b5903] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:39:19,948 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-57) [525b5903] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:39:19,948 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-57) [525b5903] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:39:19,948 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-57) [525b5903] FINISH, ConnectStoragePoolVDSCommand, log id: 4c852f2c >2012-09-02 15:39:19,948 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [525b5903] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:39:19,948 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [525b5903] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:39:20,417 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-57) [5bd52c21] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:39:20,484 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-57) [5bd52c21] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 330751c9 >2012-09-02 15:39:20,484 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [5bd52c21] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:39:20,484 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-57) [5bd52c21] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 330751c9 >2012-09-02 15:39:30,539 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:39:30,566 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-82) START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 1c160561 >2012-09-02 15:39:31,417 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-82) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:39:31,417 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-82) Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:39:31,417 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-82) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:39:31,417 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-82) FINISH, ConnectStoragePoolVDSCommand, log id: 1c160561 >2012-09-02 15:39:31,417 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:39:31,417 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:39:31,531 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-82) [14eea8ba] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:39:31,582 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-82) [14eea8ba] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 208d497d >2012-09-02 15:39:31,582 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [14eea8ba] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:39:31,582 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-82) [14eea8ba] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 208d497d >2012-09-02 15:39:41,626 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-4) hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:39:41,697 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-4) START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 79c96bca >2012-09-02 15:39:42,534 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-4) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:39:42,534 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-4) Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:39:42,534 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-4) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:39:42,534 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-4) FINISH, ConnectStoragePoolVDSCommand, log id: 79c96bca >2012-09-02 15:39:42,534 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-4) IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:39:42,534 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-4) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:39:42,609 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-4) [568674de] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:39:42,648 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-4) [568674de] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 123a1f35 >2012-09-02 15:39:42,648 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-4) [568674de] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:39:42,648 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-4) [568674de] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 123a1f35 >2012-09-02 15:39:52,687 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-12) [2b77d958] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:39:52,714 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-12) [2b77d958] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 3fecc0f5 >2012-09-02 15:39:53,552 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-12) [2b77d958] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:39:53,552 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-12) [2b77d958] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:39:53,552 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-12) [2b77d958] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:39:53,552 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-12) [2b77d958] FINISH, ConnectStoragePoolVDSCommand, log id: 3fecc0f5 >2012-09-02 15:39:53,552 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-12) [2b77d958] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:39:53,552 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-12) [2b77d958] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:39:53,720 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-12) [572b14c6] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:39:53,833 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-12) [572b14c6] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4cf5a892 >2012-09-02 15:39:53,833 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-12) [572b14c6] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:39:53,833 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-12) [572b14c6] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4cf5a892 >2012-09-02 15:40:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-23) [bedc93a] Autorecovering hosts is disabled, skipping >2012-09-02 15:40:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-23) [bedc93a] Autorecovering storage domains is disabled, skipping >2012-09-02 15:40:03,870 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-22) hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:40:03,881 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-22) START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 1d8105ad >2012-09-02 15:40:04,757 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-22) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:40:04,757 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-22) Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:40:04,757 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-22) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:40:04,757 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-22) FINISH, ConnectStoragePoolVDSCommand, log id: 1d8105ad >2012-09-02 15:40:04,757 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-22) IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:40:04,757 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-22) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:40:05,099 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-22) [3b584081] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:40:05,134 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-22) [3b584081] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6b5c50b3 >2012-09-02 15:40:05,134 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-22) [3b584081] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:40:05,134 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-22) [3b584081] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6b5c50b3 >2012-09-02 15:40:15,175 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-45) [7b580d0f] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:40:15,644 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-45) [7b580d0f] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 6b5a959b >2012-09-02 15:40:16,485 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-45) [7b580d0f] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:40:16,485 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-45) [7b580d0f] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:40:16,485 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-45) [7b580d0f] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:40:16,485 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-45) [7b580d0f] FINISH, ConnectStoragePoolVDSCommand, log id: 6b5a959b >2012-09-02 15:40:16,485 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-45) [7b580d0f] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:40:16,485 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-45) [7b580d0f] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:40:16,566 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-45) [5fc0add6] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:40:16,623 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-45) [5fc0add6] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5eb18ec8 >2012-09-02 15:40:16,624 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-45) [5fc0add6] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:40:16,624 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-45) [5fc0add6] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5eb18ec8 >2012-09-02 15:40:26,670 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:40:26,690 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-73) START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 3092f377 >2012-09-02 15:40:27,552 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-73) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:40:27,552 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-73) Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:40:27,552 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-73) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:40:27,552 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-73) FINISH, ConnectStoragePoolVDSCommand, log id: 3092f377 >2012-09-02 15:40:27,552 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:40:27,552 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:40:27,654 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-73) [ca56dc4] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:40:27,697 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-73) [ca56dc4] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 63fe36a6 >2012-09-02 15:40:27,697 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [ca56dc4] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:40:27,697 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-73) [ca56dc4] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 63fe36a6 >2012-09-02 15:40:37,735 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-81) [6454b998] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:40:37,754 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-81) [6454b998] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 43604b31 >2012-09-02 15:40:38,592 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-81) [6454b998] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:40:38,592 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-81) [6454b998] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:40:38,592 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-81) [6454b998] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:40:38,592 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-81) [6454b998] FINISH, ConnectStoragePoolVDSCommand, log id: 43604b31 >2012-09-02 15:40:38,592 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-81) [6454b998] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:40:38,592 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-81) [6454b998] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:40:38,697 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-81) [3023d06f] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:40:38,733 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-81) [3023d06f] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 15e9c164 >2012-09-02 15:40:38,733 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-81) [3023d06f] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:40:38,733 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-81) [3023d06f] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 15e9c164 >2012-09-02 15:40:48,783 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:40:48,800 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-96) START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: e633c4a >2012-09-02 15:40:49,659 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-96) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:40:49,659 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-96) Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:40:49,659 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-96) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:40:49,659 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-96) FINISH, ConnectStoragePoolVDSCommand, log id: e633c4a >2012-09-02 15:40:49,659 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:40:49,659 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:40:50,292 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-96) [239c5a4] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:40:50,327 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-96) [239c5a4] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1f172b9d >2012-09-02 15:40:50,328 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [239c5a4] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:40:50,328 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-96) [239c5a4] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1f172b9d >2012-09-02 15:41:00,362 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [63b152d2] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:41:00,443 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-5) [63b152d2] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 4b7271ca >2012-09-02 15:41:01,281 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-5) [63b152d2] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:41:01,281 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-5) [63b152d2] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:41:01,282 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-5) [63b152d2] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:41:01,282 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-5) [63b152d2] FINISH, ConnectStoragePoolVDSCommand, log id: 4b7271ca >2012-09-02 15:41:01,282 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [63b152d2] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:41:01,282 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [63b152d2] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:41:01,360 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-5) [55aad9b5] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:41:01,393 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-5) [55aad9b5] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6cc093ef >2012-09-02 15:41:01,393 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [55aad9b5] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:41:01,393 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-5) [55aad9b5] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6cc093ef >2012-09-02 15:41:11,442 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [bedc93a] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:41:11,692 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-23) [bedc93a] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 4736a5da >2012-09-02 15:41:12,821 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-23) [bedc93a] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:41:12,822 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-23) [bedc93a] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:41:12,822 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-23) [bedc93a] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:41:12,822 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-23) [bedc93a] FINISH, ConnectStoragePoolVDSCommand, log id: 4736a5da >2012-09-02 15:41:12,822 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [bedc93a] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:41:12,822 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [bedc93a] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:41:12,888 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-23) [42b68351] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:41:12,927 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-23) [42b68351] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 18b0173a >2012-09-02 15:41:12,928 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [42b68351] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:41:12,928 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-23) [42b68351] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 18b0173a >2012-09-02 15:41:22,989 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-43) hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:41:23,007 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-43) START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 1d2c73d1 >2012-09-02 15:41:23,945 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-43) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:41:23,945 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-43) Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:41:23,945 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-43) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:41:23,945 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-43) FINISH, ConnectStoragePoolVDSCommand, log id: 1d2c73d1 >2012-09-02 15:41:23,945 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-43) IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:41:23,945 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-43) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:41:24,048 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-43) [35748f38] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:41:24,109 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-43) [35748f38] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 55ac02df >2012-09-02 15:41:24,109 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-43) [35748f38] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:41:24,109 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-43) [35748f38] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 55ac02df >2012-09-02 15:41:34,151 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) [5df6ca7d] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:41:34,188 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-51) [5df6ca7d] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 5e814d88 >2012-09-02 15:41:35,047 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-51) [5df6ca7d] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:41:35,047 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-51) [5df6ca7d] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:41:35,048 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-51) [5df6ca7d] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:41:35,048 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-51) [5df6ca7d] FINISH, ConnectStoragePoolVDSCommand, log id: 5e814d88 >2012-09-02 15:41:35,048 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) [5df6ca7d] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:41:35,048 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) [5df6ca7d] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:41:35,220 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-51) [46ff88ea] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:41:35,253 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-51) [46ff88ea] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 33d8ce5f >2012-09-02 15:41:35,253 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) [46ff88ea] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:41:35,253 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-51) [46ff88ea] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 33d8ce5f >2012-09-02 15:41:45,297 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [53349194] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:41:45,313 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-71) [53349194] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 3087728c >2012-09-02 15:41:46,189 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-71) [53349194] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:41:46,189 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-71) [53349194] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:41:46,189 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-71) [53349194] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:41:46,189 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-71) [53349194] FINISH, ConnectStoragePoolVDSCommand, log id: 3087728c >2012-09-02 15:41:46,190 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [53349194] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:41:46,190 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [53349194] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:41:46,274 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-71) [89b785c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:41:46,311 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-71) [89b785c] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2c495461 >2012-09-02 15:41:46,311 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [89b785c] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:41:46,311 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-71) [89b785c] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2c495461 >2012-09-02 15:41:56,350 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-91) hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:41:56,374 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-91) START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 1c6d5f87 >2012-09-02 15:41:57,200 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-91) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:41:57,200 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-91) Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:41:57,200 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-91) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:41:57,200 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-91) FINISH, ConnectStoragePoolVDSCommand, log id: 1c6d5f87 >2012-09-02 15:41:57,200 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-91) IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:41:57,200 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-91) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:41:57,282 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-91) [303ed861] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:41:57,315 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-91) [303ed861] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 24e4dfab >2012-09-02 15:41:57,315 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-91) [303ed861] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:41:57,315 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-91) [303ed861] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 24e4dfab >2012-09-02 15:42:07,364 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) [45df5ae9] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:42:07,382 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-2) [45df5ae9] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 264e9b16 >2012-09-02 15:42:08,243 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-2) [45df5ae9] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:42:08,243 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-2) [45df5ae9] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:42:08,243 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-2) [45df5ae9] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:42:08,243 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-2) [45df5ae9] FINISH, ConnectStoragePoolVDSCommand, log id: 264e9b16 >2012-09-02 15:42:08,243 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) [45df5ae9] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:42:08,243 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) [45df5ae9] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:42:08,850 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-2) [2a8c60b0] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:42:08,969 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-2) [2a8c60b0] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 238d5812 >2012-09-02 15:42:08,970 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) [2a8c60b0] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:42:08,970 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-2) [2a8c60b0] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 238d5812 >2012-09-02 15:42:19,024 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [4d5ce074] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:42:19,161 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-17) [4d5ce074] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 1b30d649 >2012-09-02 15:42:20,006 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-17) [4d5ce074] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:42:20,359 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-17) [4d5ce074] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:42:20,360 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-17) [4d5ce074] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:42:20,360 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-17) [4d5ce074] FINISH, ConnectStoragePoolVDSCommand, log id: 1b30d649 >2012-09-02 15:42:20,360 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [4d5ce074] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:42:20,360 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [4d5ce074] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:42:20,894 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-17) [7445a7b7] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:42:20,925 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-17) [7445a7b7] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 28bdbd8a >2012-09-02 15:42:20,925 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [7445a7b7] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:42:20,925 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-17) [7445a7b7] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 28bdbd8a >2012-09-02 15:42:30,971 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-35) [6a98ae20] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:42:30,990 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-35) [6a98ae20] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 5b920c94 >2012-09-02 15:42:31,844 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-35) [6a98ae20] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:42:31,844 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-35) [6a98ae20] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:42:31,844 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-35) [6a98ae20] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:42:31,844 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-35) [6a98ae20] FINISH, ConnectStoragePoolVDSCommand, log id: 5b920c94 >2012-09-02 15:42:31,844 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-35) [6a98ae20] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:42:31,844 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-35) [6a98ae20] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:42:31,942 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-35) [6514e28e] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:42:31,986 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-35) [6514e28e] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2c2b17a1 >2012-09-02 15:42:31,987 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-35) [6514e28e] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:42:31,987 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-35) [6514e28e] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2c2b17a1 >2012-09-02 15:42:42,019 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-52) [1d401342] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:42:42,047 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-52) [1d401342] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 3e5cac57 >2012-09-02 15:42:42,861 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-52) [1d401342] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:42:42,861 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-52) [1d401342] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:42:42,861 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-52) [1d401342] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:42:42,861 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-52) [1d401342] FINISH, ConnectStoragePoolVDSCommand, log id: 3e5cac57 >2012-09-02 15:42:42,861 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-52) [1d401342] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:42:42,861 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-52) [1d401342] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:42:42,945 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-52) [25430fdb] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:42:42,988 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-52) [25430fdb] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 64a1dc95 >2012-09-02 15:42:42,988 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-52) [25430fdb] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:42:42,989 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-52) [25430fdb] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 64a1dc95 >2012-09-02 15:42:53,050 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [5bd52c21] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:42:53,075 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-57) [5bd52c21] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 40cdbd01 >2012-09-02 15:42:53,950 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-57) [5bd52c21] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:42:53,951 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-57) [5bd52c21] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:42:53,951 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-57) [5bd52c21] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:42:53,951 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-57) [5bd52c21] FINISH, ConnectStoragePoolVDSCommand, log id: 40cdbd01 >2012-09-02 15:42:53,951 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [5bd52c21] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:42:53,951 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [5bd52c21] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:42:54,038 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-57) [6af76567] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:42:54,088 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-57) [6af76567] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5930a70d >2012-09-02 15:42:54,088 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [6af76567] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:42:54,088 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-57) [6af76567] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5930a70d >2012-09-02 15:43:04,143 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) [11e497be] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:43:04,157 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-86) [11e497be] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 798e8d33 >2012-09-02 15:43:05,028 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-86) [11e497be] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:43:05,028 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-86) [11e497be] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:43:05,028 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-86) [11e497be] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:43:05,028 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-86) [11e497be] FINISH, ConnectStoragePoolVDSCommand, log id: 798e8d33 >2012-09-02 15:43:05,028 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) [11e497be] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:43:05,028 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) [11e497be] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:43:05,209 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-86) [1b72b674] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:43:05,256 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-86) [1b72b674] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 76d13958 >2012-09-02 15:43:05,256 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) [1b72b674] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:43:05,257 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-86) [1b72b674] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 76d13958 >2012-09-02 15:43:15,299 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-99) hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:43:15,311 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-99) START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 66604fe4 >2012-09-02 15:43:16,170 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-99) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:43:16,170 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-99) Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:43:16,170 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-99) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:43:16,170 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-99) FINISH, ConnectStoragePoolVDSCommand, log id: 66604fe4 >2012-09-02 15:43:16,170 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-99) IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:43:16,170 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-99) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:43:16,259 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-99) [5b20d4d] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:43:16,303 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-99) [5b20d4d] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5ae6846d >2012-09-02 15:43:16,303 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-99) [5b20d4d] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:43:16,303 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-99) [5b20d4d] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5ae6846d >2012-09-02 15:43:26,349 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-15) [d40d6cd] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:43:26,364 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-15) [d40d6cd] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 15f44dcb >2012-09-02 15:43:27,229 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-15) [d40d6cd] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:43:27,229 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-15) [d40d6cd] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:43:27,229 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-15) [d40d6cd] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:43:27,229 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-15) [d40d6cd] FINISH, ConnectStoragePoolVDSCommand, log id: 15f44dcb >2012-09-02 15:43:27,229 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-15) [d40d6cd] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:43:27,229 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-15) [d40d6cd] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:43:27,296 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-15) [5147a4f8] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:43:27,361 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-15) [5147a4f8] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 26fe28b1 >2012-09-02 15:43:27,361 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-15) [5147a4f8] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:43:27,361 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-15) [5147a4f8] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 26fe28b1 >2012-09-02 15:43:37,398 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-28) [45c77945] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:43:37,601 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-28) [45c77945] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 1366d9a5 >2012-09-02 15:43:38,438 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-28) [45c77945] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:43:38,438 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-28) [45c77945] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:43:38,438 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-28) [45c77945] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:43:38,438 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-28) [45c77945] FINISH, ConnectStoragePoolVDSCommand, log id: 1366d9a5 >2012-09-02 15:43:38,438 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-28) [45c77945] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:43:38,438 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-28) [45c77945] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:43:38,521 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-28) [715bed64] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:43:38,562 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-28) [715bed64] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 114bdaaa >2012-09-02 15:43:38,562 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-28) [715bed64] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:43:38,562 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-28) [715bed64] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 114bdaaa >2012-09-02 15:43:48,597 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-45) [5fc0add6] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:43:48,611 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-45) [5fc0add6] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 7f5a0691 >2012-09-02 15:43:49,457 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-45) [5fc0add6] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:43:49,457 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-45) [5fc0add6] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:43:49,457 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-45) [5fc0add6] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:43:49,457 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-45) [5fc0add6] FINISH, ConnectStoragePoolVDSCommand, log id: 7f5a0691 >2012-09-02 15:43:49,457 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-45) [5fc0add6] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:43:49,457 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-45) [5fc0add6] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:43:49,530 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-45) [4a22f461] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:43:49,572 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-45) [4a22f461] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 31c7f >2012-09-02 15:43:49,572 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-45) [4a22f461] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:43:49,572 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-45) [4a22f461] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 31c7f >2012-09-02 15:43:59,608 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [ca56dc4] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:43:59,620 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-73) [ca56dc4] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 50747662 >2012-09-02 15:44:00,471 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-73) [ca56dc4] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:44:00,471 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-73) [ca56dc4] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:44:00,471 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-73) [ca56dc4] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:44:00,471 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-73) [ca56dc4] FINISH, ConnectStoragePoolVDSCommand, log id: 50747662 >2012-09-02 15:44:00,471 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [ca56dc4] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:44:00,471 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [ca56dc4] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:44:00,591 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-73) [49f1e971] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:44:00,622 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-73) [49f1e971] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6921148e >2012-09-02 15:44:00,622 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [49f1e971] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:44:00,622 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-73) [49f1e971] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6921148e >2012-09-02 15:44:10,656 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:44:10,671 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-84) START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 49b39bf >2012-09-02 15:44:11,507 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-84) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:44:11,507 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-84) Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:44:11,507 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-84) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:44:11,507 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-84) FINISH, ConnectStoragePoolVDSCommand, log id: 49b39bf >2012-09-02 15:44:11,507 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:44:11,507 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:44:11,587 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-84) [17f32511] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:44:11,657 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-84) [17f32511] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 27e39797 >2012-09-02 15:44:11,657 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [17f32511] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:44:11,657 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-84) [17f32511] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 27e39797 >2012-09-02 15:44:21,692 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [536973ff] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:44:22,032 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-94) [536973ff] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 64f163ef >2012-09-02 15:44:22,879 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-94) [536973ff] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:44:22,880 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-94) [536973ff] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:44:22,880 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-94) [536973ff] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:44:22,880 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-94) [536973ff] FINISH, ConnectStoragePoolVDSCommand, log id: 64f163ef >2012-09-02 15:44:22,880 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [536973ff] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:44:22,880 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [536973ff] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:44:23,458 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-94) [2385966b] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:44:23,502 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-94) [2385966b] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5698c730 >2012-09-02 15:44:23,502 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [2385966b] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:44:23,502 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-94) [2385966b] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5698c730 >2012-09-02 15:44:33,538 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-9) [6bd9f19d] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:44:33,569 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-9) [6bd9f19d] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 4cc600a1 >2012-09-02 15:44:34,414 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-9) [6bd9f19d] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:44:34,414 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-9) [6bd9f19d] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:44:34,414 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-9) [6bd9f19d] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:44:34,414 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-9) [6bd9f19d] FINISH, ConnectStoragePoolVDSCommand, log id: 4cc600a1 >2012-09-02 15:44:34,415 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-9) [6bd9f19d] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:44:34,415 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-9) [6bd9f19d] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:44:34,499 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-9) [7fde1f] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:44:34,535 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-9) [7fde1f] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 39f5481d >2012-09-02 15:44:34,535 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-9) [7fde1f] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:44:34,535 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-9) [7fde1f] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 39f5481d >2012-09-02 15:44:44,576 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-26) hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:44:44,971 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-26) START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 3ec131e0 >2012-09-02 15:44:45,830 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-26) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:44:45,830 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-26) Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:44:45,830 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-26) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:44:45,831 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-26) FINISH, ConnectStoragePoolVDSCommand, log id: 3ec131e0 >2012-09-02 15:44:45,831 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-26) IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:44:45,831 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-26) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:44:45,913 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-26) [621177bf] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:44:45,954 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-26) [621177bf] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1591247d >2012-09-02 15:44:45,954 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-26) [621177bf] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:44:45,954 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-26) [621177bf] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1591247d >2012-09-02 15:44:55,993 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-47) hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:44:56,008 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-47) START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 762dd9a7 >2012-09-02 15:44:56,858 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:44:56,859 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-47) Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:44:56,859 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-47) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:44:56,859 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-47) FINISH, ConnectStoragePoolVDSCommand, log id: 762dd9a7 >2012-09-02 15:44:56,859 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-47) IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:44:56,859 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-47) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:44:56,926 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-47) [7061330f] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:44:56,962 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-47) [7061330f] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 10f2b74f >2012-09-02 15:44:56,962 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-47) [7061330f] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:44:56,962 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-47) [7061330f] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 10f2b74f >2012-09-02 15:45:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-45) [4a22f461] Autorecovering hosts is disabled, skipping >2012-09-02 15:45:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-45) [4a22f461] Autorecovering storage domains is disabled, skipping >2012-09-02 15:45:07,001 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:45:07,021 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-65) START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 3d653598 >2012-09-02 15:45:07,887 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-65) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:45:07,887 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-65) Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:45:07,887 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-65) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:45:07,887 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-65) FINISH, ConnectStoragePoolVDSCommand, log id: 3d653598 >2012-09-02 15:45:07,887 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:45:07,887 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:45:09,098 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-65) [ffaf13b] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:45:09,170 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-65) [ffaf13b] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7b9a9529 >2012-09-02 15:45:09,170 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) [ffaf13b] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:45:09,170 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-65) [ffaf13b] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7b9a9529 >2012-09-02 15:45:19,211 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-80) [3692be71] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:45:19,409 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-80) [3692be71] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 40362a3c >2012-09-02 15:45:20,272 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-80) [3692be71] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:45:20,272 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-80) [3692be71] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:45:20,272 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-80) [3692be71] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:45:20,272 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-80) [3692be71] FINISH, ConnectStoragePoolVDSCommand, log id: 40362a3c >2012-09-02 15:45:20,272 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-80) [3692be71] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:45:20,272 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-80) [3692be71] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:45:20,881 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-80) [154a4686] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:45:20,953 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-80) [154a4686] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7a73de9d >2012-09-02 15:45:20,953 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-80) [154a4686] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:45:20,953 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-80) [154a4686] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7a73de9d >2012-09-02 15:45:31,006 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [100a6607] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:45:31,022 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-87) [100a6607] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 481053ee >2012-09-02 15:45:31,885 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-87) [100a6607] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:45:31,885 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-87) [100a6607] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:45:31,885 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-87) [100a6607] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:45:31,885 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-87) [100a6607] FINISH, ConnectStoragePoolVDSCommand, log id: 481053ee >2012-09-02 15:45:31,885 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [100a6607] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:45:31,885 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [100a6607] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:45:31,963 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-87) [62351dda] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:45:32,011 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-87) [62351dda] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 455d6db5 >2012-09-02 15:45:32,011 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [62351dda] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:45:32,011 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-87) [62351dda] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 455d6db5 >2012-09-02 15:45:42,053 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [97db800] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:45:42,077 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-8) [97db800] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 5a48abbd >2012-09-02 15:45:42,918 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-8) [97db800] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:45:42,918 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-8) [97db800] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:45:42,918 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-8) [97db800] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:45:42,918 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-8) [97db800] FINISH, ConnectStoragePoolVDSCommand, log id: 5a48abbd >2012-09-02 15:45:42,918 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [97db800] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:45:42,918 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [97db800] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:45:43,090 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-8) [52fbf215] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:45:43,152 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-8) [52fbf215] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 68d32338 >2012-09-02 15:45:43,152 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [52fbf215] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:45:43,152 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-8) [52fbf215] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 68d32338 >2012-09-02 15:45:53,204 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) [ad5ebcb] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:45:53,240 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-21) [ad5ebcb] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 91abdee >2012-09-02 15:45:54,086 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-21) [ad5ebcb] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:45:54,086 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-21) [ad5ebcb] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:45:54,086 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-21) [ad5ebcb] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:45:54,086 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-21) [ad5ebcb] FINISH, ConnectStoragePoolVDSCommand, log id: 91abdee >2012-09-02 15:45:54,086 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) [ad5ebcb] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:45:54,086 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) [ad5ebcb] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:45:54,197 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-21) [2adea929] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:45:54,242 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-21) [2adea929] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4e96d6fb >2012-09-02 15:45:54,242 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) [2adea929] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:45:54,242 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-21) [2adea929] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4e96d6fb >2012-09-02 15:46:04,289 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [48816517] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:46:04,332 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-42) [48816517] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 52721175 >2012-09-02 15:46:04,955 INFO [org.ovirt.engine.core.bll.LoginAdminUserCommand] (ajp-/127.0.0.1:8009-2) Checking if user admin@internal is an admin, result true >2012-09-02 15:46:04,955 INFO [org.ovirt.engine.core.bll.LoginAdminUserCommand] (ajp-/127.0.0.1:8009-2) Running command: LoginAdminUserCommand internal: false. >2012-09-02 15:46:05,163 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-42) [48816517] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:46:05,163 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-42) [48816517] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:46:05,163 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-42) [48816517] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:46:05,163 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-42) [48816517] FINISH, ConnectStoragePoolVDSCommand, log id: 52721175 >2012-09-02 15:46:05,164 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [48816517] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:46:05,164 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [48816517] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:46:05,243 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-42) [1cf6d845] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:46:05,285 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-42) [1cf6d845] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 33977144 >2012-09-02 15:46:05,285 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [1cf6d845] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:46:05,285 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-42) [1cf6d845] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 33977144 >2012-09-02 15:46:06,230 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8009-5) calling GetConfigurationValueQuery (CustomPublicConfig_AppsWebSite) with null version, using default general for version >2012-09-02 15:46:15,330 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-34) hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:46:15,352 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-34) START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 154a645d >2012-09-02 15:46:16,195 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-34) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:46:16,440 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-34) Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:46:16,440 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-34) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:46:16,440 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-34) FINISH, ConnectStoragePoolVDSCommand, log id: 154a645d >2012-09-02 15:46:16,440 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-34) IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:46:16,440 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-34) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:46:16,609 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-34) [39d95b35] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:46:16,667 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-34) [39d95b35] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1939932b >2012-09-02 15:46:16,667 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-34) [39d95b35] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:46:16,667 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-34) [39d95b35] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1939932b >2012-09-02 15:46:26,706 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [27a95d66] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:46:26,757 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-69) [27a95d66] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 68ee5574 >2012-09-02 15:46:27,599 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-69) [27a95d66] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:46:27,599 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-69) [27a95d66] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:46:27,599 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-69) [27a95d66] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:46:27,599 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-69) [27a95d66] FINISH, ConnectStoragePoolVDSCommand, log id: 68ee5574 >2012-09-02 15:46:27,599 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [27a95d66] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:46:27,599 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [27a95d66] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:46:27,694 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-69) [39fd24f3] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:46:27,747 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-69) [39fd24f3] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7acab5c6 >2012-09-02 15:46:27,747 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [39fd24f3] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:46:27,748 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-69) [39fd24f3] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7acab5c6 >2012-09-02 15:46:37,789 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [2863ec16] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:46:37,806 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-83) [2863ec16] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 25a0067d >2012-09-02 15:46:38,667 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-83) [2863ec16] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:46:38,667 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-83) [2863ec16] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:46:38,667 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-83) [2863ec16] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:46:38,667 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-83) [2863ec16] FINISH, ConnectStoragePoolVDSCommand, log id: 25a0067d >2012-09-02 15:46:38,667 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [2863ec16] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:46:38,667 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [2863ec16] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:46:38,739 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-83) [6bd8b57] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:46:38,764 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-83) [6bd8b57] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 448a16b2 >2012-09-02 15:46:38,764 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [6bd8b57] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:46:38,764 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-83) [6bd8b57] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 448a16b2 >2012-09-02 15:46:48,799 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-100) [5bcc0093] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:46:48,836 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-100) [5bcc0093] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 1dbefc1e >2012-09-02 15:46:49,693 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-100) [5bcc0093] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:46:49,693 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-100) [5bcc0093] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:46:49,693 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-100) [5bcc0093] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:46:49,693 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-100) [5bcc0093] FINISH, ConnectStoragePoolVDSCommand, log id: 1dbefc1e >2012-09-02 15:46:49,693 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-100) [5bcc0093] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:46:49,694 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-100) [5bcc0093] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:46:50,370 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-100) [6dae079f] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:46:50,500 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-100) [6dae079f] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 34014f18 >2012-09-02 15:46:50,500 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-100) [6dae079f] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:46:50,500 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-100) [6dae079f] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 34014f18 >2012-09-02 15:47:00,576 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-19) [29d285ee] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:47:00,598 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-19) [29d285ee] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 56ca6964 >2012-09-02 15:47:01,481 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-19) [29d285ee] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:47:01,481 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-19) [29d285ee] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:47:01,481 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-19) [29d285ee] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:47:01,481 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-19) [29d285ee] FINISH, ConnectStoragePoolVDSCommand, log id: 56ca6964 >2012-09-02 15:47:01,481 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-19) [29d285ee] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:47:01,482 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-19) [29d285ee] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:47:01,562 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-19) [229837ce] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:47:01,594 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-19) [229837ce] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1f08bfc1 >2012-09-02 15:47:01,594 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-19) [229837ce] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:47:01,594 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-19) [229837ce] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1f08bfc1 >2012-09-02 15:47:11,642 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-36) hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:47:11,658 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-36) START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 2ed9660d >2012-09-02 15:47:12,532 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-36) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:47:12,532 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-36) Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:47:12,533 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-36) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:47:12,533 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-36) FINISH, ConnectStoragePoolVDSCommand, log id: 2ed9660d >2012-09-02 15:47:12,533 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-36) IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:47:12,533 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-36) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:47:12,811 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-36) [6c7e8a86] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:47:12,848 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-36) [6c7e8a86] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 776c1bf8 >2012-09-02 15:47:12,849 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-36) [6c7e8a86] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:47:12,849 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-36) [6c7e8a86] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 776c1bf8 >2012-09-02 15:47:22,881 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [308ea3f7] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:47:22,915 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-50) [308ea3f7] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 7fad0efb >2012-09-02 15:47:23,767 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-50) [308ea3f7] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:47:23,767 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-50) [308ea3f7] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:47:23,767 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-50) [308ea3f7] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:47:23,767 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-50) [308ea3f7] FINISH, ConnectStoragePoolVDSCommand, log id: 7fad0efb >2012-09-02 15:47:23,767 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [308ea3f7] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:47:23,767 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [308ea3f7] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:47:23,972 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-50) [6dc69ea2] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:47:24,021 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-50) [6dc69ea2] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5952c260 >2012-09-02 15:47:24,021 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [6dc69ea2] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:47:24,021 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-50) [6dc69ea2] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5952c260 >2012-09-02 15:47:34,060 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [49f1e971] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:47:34,077 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-73) [49f1e971] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 7f5ca262 >2012-09-02 15:47:34,924 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-73) [49f1e971] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:47:34,924 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-73) [49f1e971] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:47:34,924 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-73) [49f1e971] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:47:34,924 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-73) [49f1e971] FINISH, ConnectStoragePoolVDSCommand, log id: 7f5ca262 >2012-09-02 15:47:34,924 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [49f1e971] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:47:34,924 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [49f1e971] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:47:35,042 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-73) [3b94000c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:47:35,084 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-73) [3b94000c] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 451c661 >2012-09-02 15:47:35,084 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [3b94000c] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:47:35,084 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-73) [3b94000c] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 451c661 >2012-09-02 15:47:45,134 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-89) hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:47:45,152 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-89) START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 16984338 >2012-09-02 15:47:45,991 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-89) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:47:45,991 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-89) Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:47:45,991 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-89) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:47:45,991 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-89) FINISH, ConnectStoragePoolVDSCommand, log id: 16984338 >2012-09-02 15:47:45,991 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-89) IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:47:45,991 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-89) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:47:46,105 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-89) [5fb3a86f] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:47:46,166 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-89) [5fb3a86f] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 11c8f9a4 >2012-09-02 15:47:46,166 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-89) [5fb3a86f] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:47:46,166 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-89) [5fb3a86f] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 11c8f9a4 >2012-09-02 15:47:56,206 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-78) [6a1d44c4] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:47:56,258 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-78) [6a1d44c4] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: e82e29b >2012-09-02 15:47:57,115 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-78) [6a1d44c4] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:47:57,116 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-78) [6a1d44c4] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:47:57,116 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-78) [6a1d44c4] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:47:57,116 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-78) [6a1d44c4] FINISH, ConnectStoragePoolVDSCommand, log id: e82e29b >2012-09-02 15:47:57,116 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-78) [6a1d44c4] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:47:57,116 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-78) [6a1d44c4] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:47:57,187 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-78) [1081755f] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:47:57,222 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-78) [1081755f] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 165cb2c9 >2012-09-02 15:47:57,223 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-78) [1081755f] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:47:57,223 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-78) [1081755f] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 165cb2c9 >2012-09-02 15:48:07,271 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-12) [572b14c6] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:48:07,288 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-12) [572b14c6] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 6d49e699 >2012-09-02 15:48:08,131 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-12) [572b14c6] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:48:08,131 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-12) [572b14c6] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:48:08,131 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-12) [572b14c6] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:48:08,131 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-12) [572b14c6] FINISH, ConnectStoragePoolVDSCommand, log id: 6d49e699 >2012-09-02 15:48:08,131 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-12) [572b14c6] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:48:08,132 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-12) [572b14c6] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:48:09,288 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-12) [5064a4c0] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:48:09,325 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-12) [5064a4c0] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4ae5d7e8 >2012-09-02 15:48:09,325 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-12) [5064a4c0] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:48:09,325 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-12) [5064a4c0] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4ae5d7e8 >2012-09-02 15:48:19,373 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-28) [715bed64] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:48:19,471 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-28) [715bed64] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 70c7398c >2012-09-02 15:48:20,312 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-28) [715bed64] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:48:20,313 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-28) [715bed64] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:48:20,313 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-28) [715bed64] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:48:20,313 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-28) [715bed64] FINISH, ConnectStoragePoolVDSCommand, log id: 70c7398c >2012-09-02 15:48:20,313 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-28) [715bed64] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:48:20,313 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-28) [715bed64] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:48:21,854 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-28) [164d9fbc] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:48:21,895 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-28) [164d9fbc] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1f4409a4 >2012-09-02 15:48:21,895 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-28) [164d9fbc] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:48:21,895 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-28) [164d9fbc] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1f4409a4 >2012-09-02 15:48:31,948 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-49) hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:48:31,975 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-49) START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 2c37a927 >2012-09-02 15:48:32,828 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:48:32,828 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-49) Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:48:32,828 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-49) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:48:32,828 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-49) FINISH, ConnectStoragePoolVDSCommand, log id: 2c37a927 >2012-09-02 15:48:32,828 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-49) IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:48:32,828 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-49) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:48:32,899 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-49) [767e356a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:48:32,928 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-49) [767e356a] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: ae1cb0a >2012-09-02 15:48:32,928 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-49) [767e356a] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:48:32,928 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-49) [767e356a] FINISH, MarkPoolInReconstructModeVDSCommand, log id: ae1cb0a >2012-09-02 15:48:42,968 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [73165b38] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:48:42,987 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-59) [73165b38] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 21270d27 >2012-09-02 15:48:43,849 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-59) [73165b38] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:48:43,849 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-59) [73165b38] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:48:43,849 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-59) [73165b38] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:48:43,849 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-59) [73165b38] FINISH, ConnectStoragePoolVDSCommand, log id: 21270d27 >2012-09-02 15:48:43,849 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [73165b38] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:48:43,849 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [73165b38] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:48:44,673 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-59) [4f91000b] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:48:45,173 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-59) [4f91000b] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 40c7246c >2012-09-02 15:48:45,173 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [4f91000b] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:48:45,173 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-59) [4f91000b] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 40c7246c >2012-09-02 15:48:55,337 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [14eea8ba] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:48:55,363 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-82) [14eea8ba] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 434aac73 >2012-09-02 15:48:56,333 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-82) [14eea8ba] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:48:56,333 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-82) [14eea8ba] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:48:56,334 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-82) [14eea8ba] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:48:56,334 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-82) [14eea8ba] FINISH, ConnectStoragePoolVDSCommand, log id: 434aac73 >2012-09-02 15:48:56,334 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [14eea8ba] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:48:56,334 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [14eea8ba] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:48:56,469 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-82) [f21968f] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:48:56,557 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-82) [f21968f] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6dfc201 >2012-09-02 15:48:56,557 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [f21968f] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:48:56,557 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-82) [f21968f] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6dfc201 >2012-09-02 15:49:06,614 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-4) [568674de] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:49:07,103 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-4) [568674de] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 1a6fa3bc >2012-09-02 15:49:08,062 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-4) [568674de] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:49:08,063 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-4) [568674de] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:49:08,063 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-4) [568674de] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:49:08,063 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-4) [568674de] FINISH, ConnectStoragePoolVDSCommand, log id: 1a6fa3bc >2012-09-02 15:49:08,063 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-4) [568674de] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:49:08,063 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-4) [568674de] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:49:08,712 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-4) [1b9aaac2] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:49:08,787 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-4) [1b9aaac2] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 43cf7243 >2012-09-02 15:49:08,787 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-4) [1b9aaac2] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:49:08,787 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-4) [1b9aaac2] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 43cf7243 >2012-09-02 15:49:18,831 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-18) hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:49:18,896 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-18) START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 614e594e >2012-09-02 15:49:19,740 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-18) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:49:19,741 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-18) Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:49:19,741 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-18) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:49:19,741 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-18) FINISH, ConnectStoragePoolVDSCommand, log id: 614e594e >2012-09-02 15:49:19,741 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-18) IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:49:19,741 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-18) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:49:20,178 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-18) [64c55214] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:49:20,264 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-18) [64c55214] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 266272f2 >2012-09-02 15:49:20,264 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-18) [64c55214] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:49:20,264 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-18) [64c55214] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 266272f2 >2012-09-02 15:49:30,319 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-26) [621177bf] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:49:30,412 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-26) [621177bf] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 4e2795aa >2012-09-02 15:49:31,271 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-26) [621177bf] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:49:31,271 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-26) [621177bf] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:49:31,271 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-26) [621177bf] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:49:31,271 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-26) [621177bf] FINISH, ConnectStoragePoolVDSCommand, log id: 4e2795aa >2012-09-02 15:49:31,271 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-26) [621177bf] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:49:31,271 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-26) [621177bf] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:49:31,556 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-26) [285a1abc] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:49:31,626 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-26) [285a1abc] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 48f8341d >2012-09-02 15:49:31,626 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-26) [285a1abc] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:49:31,626 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-26) [285a1abc] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 48f8341d >2012-09-02 15:49:41,680 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [7509bad3] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:49:41,703 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-48) [7509bad3] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 6d21269b >2012-09-02 15:49:42,564 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-48) [7509bad3] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:49:42,564 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-48) [7509bad3] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:49:42,564 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-48) [7509bad3] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:49:42,564 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-48) [7509bad3] FINISH, ConnectStoragePoolVDSCommand, log id: 6d21269b >2012-09-02 15:49:42,564 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [7509bad3] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:49:42,564 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [7509bad3] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:49:42,703 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-48) [6c48fb52] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:49:42,746 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-48) [6c48fb52] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 53ec2cef >2012-09-02 15:49:42,746 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [6c48fb52] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:49:42,746 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-48) [6c48fb52] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 53ec2cef >2012-09-02 15:49:52,797 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) [3942714e] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:49:52,922 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-55) [3942714e] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 3fc6d9c >2012-09-02 15:49:53,764 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-55) [3942714e] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:49:53,764 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-55) [3942714e] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:49:53,764 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-55) [3942714e] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:49:53,764 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-55) [3942714e] FINISH, ConnectStoragePoolVDSCommand, log id: 3fc6d9c >2012-09-02 15:49:53,764 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) [3942714e] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:49:53,765 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) [3942714e] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:49:54,326 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-55) [2e050512] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:49:54,378 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-55) [2e050512] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 75fbb449 >2012-09-02 15:49:54,378 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) [2e050512] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:49:54,378 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-55) [2e050512] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 75fbb449 >2012-09-02 15:50:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-79) Autorecovering hosts is disabled, skipping >2012-09-02 15:50:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-79) Autorecovering storage domains is disabled, skipping >2012-09-02 15:50:04,425 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-81) [3023d06f] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:50:04,442 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-81) [3023d06f] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 76293cf5 >2012-09-02 15:50:05,307 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-81) [3023d06f] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:50:05,307 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-81) [3023d06f] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:50:05,307 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-81) [3023d06f] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:50:05,307 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-81) [3023d06f] FINISH, ConnectStoragePoolVDSCommand, log id: 76293cf5 >2012-09-02 15:50:05,307 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-81) [3023d06f] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:50:05,307 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-81) [3023d06f] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:50:05,402 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-81) [3dd1f5ba] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:50:05,459 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-81) [3dd1f5ba] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 594b29d4 >2012-09-02 15:50:05,459 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-81) [3dd1f5ba] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:50:05,459 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-81) [3dd1f5ba] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 594b29d4 >2012-09-02 15:50:15,495 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [2385966b] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:50:15,512 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-94) [2385966b] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 5993797 >2012-09-02 15:50:16,368 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-94) [2385966b] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:50:16,368 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-94) [2385966b] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:50:16,368 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-94) [2385966b] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:50:16,368 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-94) [2385966b] FINISH, ConnectStoragePoolVDSCommand, log id: 5993797 >2012-09-02 15:50:16,368 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [2385966b] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:50:16,368 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [2385966b] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:50:16,444 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-94) [30f99192] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:50:16,492 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-94) [30f99192] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 144164c3 >2012-09-02 15:50:16,492 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [30f99192] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:50:16,492 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-94) [30f99192] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 144164c3 >2012-09-02 15:50:26,538 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-9) [7fde1f] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:50:26,554 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-9) [7fde1f] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 9b28930 >2012-09-02 15:50:27,405 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-9) [7fde1f] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:50:27,405 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-9) [7fde1f] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:50:27,405 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-9) [7fde1f] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:50:27,405 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-9) [7fde1f] FINISH, ConnectStoragePoolVDSCommand, log id: 9b28930 >2012-09-02 15:50:27,405 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-9) [7fde1f] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:50:27,405 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-9) [7fde1f] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:50:27,536 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-9) [69982a55] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:50:27,566 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-9) [69982a55] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3284089e >2012-09-02 15:50:27,566 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-9) [69982a55] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:50:27,566 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-9) [69982a55] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3284089e >2012-09-02 15:50:37,633 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [42b68351] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:50:37,655 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-23) [42b68351] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 7fac315d >2012-09-02 15:50:38,532 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-23) [42b68351] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:50:38,532 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-23) [42b68351] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:50:38,532 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-23) [42b68351] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:50:38,532 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-23) [42b68351] FINISH, ConnectStoragePoolVDSCommand, log id: 7fac315d >2012-09-02 15:50:38,532 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [42b68351] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:50:38,532 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [42b68351] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:50:38,652 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-23) [592daca9] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:50:39,149 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-23) [592daca9] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 45fc8f16 >2012-09-02 15:50:39,149 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [592daca9] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:50:39,149 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-23) [592daca9] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 45fc8f16 >2012-09-02 15:50:49,188 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [195448f7] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:50:49,381 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-38) [195448f7] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 7178d686 >2012-09-02 15:50:50,254 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-38) [195448f7] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:50:50,254 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-38) [195448f7] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:50:50,254 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-38) [195448f7] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:50:50,254 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-38) [195448f7] FINISH, ConnectStoragePoolVDSCommand, log id: 7178d686 >2012-09-02 15:50:50,254 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [195448f7] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:50:50,254 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [195448f7] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:50:50,444 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-38) [48cc8c2f] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:50:50,473 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-38) [48cc8c2f] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2085e4b0 >2012-09-02 15:50:50,473 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [48cc8c2f] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:50:50,473 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-38) [48cc8c2f] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2085e4b0 >2012-09-02 15:51:00,526 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-62) hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:51:00,557 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-62) START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 6e2c2985 >2012-09-02 15:51:01,439 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-62) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:51:01,439 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-62) Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:51:01,439 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-62) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:51:01,439 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-62) FINISH, ConnectStoragePoolVDSCommand, log id: 6e2c2985 >2012-09-02 15:51:01,439 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-62) IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:51:01,439 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-62) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:51:01,508 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-62) [3ea43f70] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:51:01,543 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-62) [3ea43f70] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 97141e6 >2012-09-02 15:51:01,544 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-62) [3ea43f70] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:51:01,544 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-62) [3ea43f70] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 97141e6 >2012-09-02 15:51:11,593 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-64) [73b5feec] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:51:11,609 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-64) [73b5feec] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 51f6a4b5 >2012-09-02 15:51:12,484 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-64) [73b5feec] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:51:12,484 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-64) [73b5feec] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:51:12,484 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-64) [73b5feec] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:51:12,484 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-64) [73b5feec] FINISH, ConnectStoragePoolVDSCommand, log id: 51f6a4b5 >2012-09-02 15:51:12,484 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-64) [73b5feec] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:51:12,484 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-64) [73b5feec] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:51:12,595 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-64) [5bdf963c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:51:12,632 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-64) [5bdf963c] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 51ec591f >2012-09-02 15:51:12,633 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-64) [5bdf963c] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:51:12,633 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-64) [5bdf963c] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 51ec591f >2012-09-02 15:51:22,668 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [6bd8b57] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:51:22,685 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-83) [6bd8b57] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 4dc7c065 >2012-09-02 15:51:23,554 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-83) [6bd8b57] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:51:23,554 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-83) [6bd8b57] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:51:23,554 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-83) [6bd8b57] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:51:23,554 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-83) [6bd8b57] FINISH, ConnectStoragePoolVDSCommand, log id: 4dc7c065 >2012-09-02 15:51:23,554 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [6bd8b57] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:51:23,554 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [6bd8b57] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:51:23,685 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-83) [f848894] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:51:23,745 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-83) [f848894] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2b03a1fe >2012-09-02 15:51:23,745 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [f848894] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:51:23,745 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-83) [f848894] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2b03a1fe >2012-09-02 15:51:33,797 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-99) [5b20d4d] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:51:33,814 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-99) [5b20d4d] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 41905790 >2012-09-02 15:51:34,669 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-99) [5b20d4d] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:51:34,669 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-99) [5b20d4d] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:51:34,669 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-99) [5b20d4d] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:51:34,669 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-99) [5b20d4d] FINISH, ConnectStoragePoolVDSCommand, log id: 41905790 >2012-09-02 15:51:34,670 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-99) [5b20d4d] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:51:34,670 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-99) [5b20d4d] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:51:35,080 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-99) [d9b82ea] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:51:35,129 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-99) [d9b82ea] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7de85f62 >2012-09-02 15:51:35,129 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-99) [d9b82ea] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:51:35,129 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-99) [d9b82ea] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7de85f62 >2012-09-02 15:51:45,193 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:51:45,217 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-27) START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 7776166 >2012-09-02 15:51:46,048 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-27) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:51:46,048 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-27) Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:51:46,048 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-27) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:51:46,048 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-27) FINISH, ConnectStoragePoolVDSCommand, log id: 7776166 >2012-09-02 15:51:46,048 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:51:46,048 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:51:46,131 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-27) [a89e0e7] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:51:46,171 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-27) [a89e0e7] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 67332e41 >2012-09-02 15:51:46,171 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [a89e0e7] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:51:46,171 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-27) [a89e0e7] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 67332e41 >2012-09-02 15:51:56,229 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [aa7db89] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:51:56,256 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-39) [aa7db89] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 35c19d89 >2012-09-02 15:51:57,110 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-39) [aa7db89] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:51:57,111 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-39) [aa7db89] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:51:57,111 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-39) [aa7db89] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:51:57,111 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-39) [aa7db89] FINISH, ConnectStoragePoolVDSCommand, log id: 35c19d89 >2012-09-02 15:51:57,111 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [aa7db89] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:51:57,111 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [aa7db89] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:51:57,209 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-39) [4fc83cbd] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:51:57,246 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-39) [4fc83cbd] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1911bc0d >2012-09-02 15:51:57,246 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [4fc83cbd] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:51:57,246 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-39) [4fc83cbd] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1911bc0d >2012-09-02 15:52:07,285 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-52) [25430fdb] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:52:07,301 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-52) [25430fdb] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 42a9c23 >2012-09-02 15:52:08,166 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-52) [25430fdb] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:52:08,167 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-52) [25430fdb] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:52:08,167 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-52) [25430fdb] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:52:08,167 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-52) [25430fdb] FINISH, ConnectStoragePoolVDSCommand, log id: 42a9c23 >2012-09-02 15:52:08,167 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-52) [25430fdb] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:52:08,167 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-52) [25430fdb] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:52:08,233 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-52) [570552a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:52:08,265 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-52) [570552a] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5c6b0b33 >2012-09-02 15:52:08,265 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-52) [570552a] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:52:08,265 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-52) [570552a] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5c6b0b33 >2012-09-02 15:52:18,499 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:52:18,515 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-72) START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: fd88b43 >2012-09-02 15:52:19,368 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-72) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:52:19,368 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-72) Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:52:19,368 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-72) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:52:19,368 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-72) FINISH, ConnectStoragePoolVDSCommand, log id: fd88b43 >2012-09-02 15:52:19,368 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:52:19,368 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:52:19,451 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-72) [2ab07b12] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:52:19,508 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-72) [2ab07b12] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 28f46cac >2012-09-02 15:52:19,508 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [2ab07b12] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:52:19,508 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-72) [2ab07b12] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 28f46cac >2012-09-02 15:52:29,577 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [df2e2a7] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:52:29,597 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-77) [df2e2a7] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 64167b1b >2012-09-02 15:52:30,426 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-77) [df2e2a7] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:52:30,426 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-77) [df2e2a7] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:52:30,426 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-77) [df2e2a7] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:52:30,426 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-77) [df2e2a7] FINISH, ConnectStoragePoolVDSCommand, log id: 64167b1b >2012-09-02 15:52:30,426 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [df2e2a7] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:52:30,426 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [df2e2a7] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:52:30,552 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-77) [71797221] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:52:30,643 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-77) [71797221] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 252ab0e8 >2012-09-02 15:52:30,643 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [71797221] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:52:30,643 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-77) [71797221] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 252ab0e8 >2012-09-02 15:52:40,681 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [5ed2c82c] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:52:40,697 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-97) [5ed2c82c] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 87cf7da >2012-09-02 15:52:41,576 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-97) [5ed2c82c] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:52:41,576 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-97) [5ed2c82c] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:52:41,576 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-97) [5ed2c82c] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:52:41,576 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-97) [5ed2c82c] FINISH, ConnectStoragePoolVDSCommand, log id: 87cf7da >2012-09-02 15:52:41,576 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [5ed2c82c] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:52:41,576 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [5ed2c82c] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:52:41,885 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-97) [55a6af91] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:52:41,943 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-97) [55a6af91] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 62cb69d9 >2012-09-02 15:52:41,943 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [55a6af91] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:52:41,943 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-97) [55a6af91] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 62cb69d9 >2012-09-02 15:52:52,010 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:52:52,030 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-16) START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 47c5cdfd >2012-09-02 15:52:52,893 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-16) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:52:52,893 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-16) Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:52:52,893 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-16) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:52:52,894 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-16) FINISH, ConnectStoragePoolVDSCommand, log id: 47c5cdfd >2012-09-02 15:52:52,894 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 15:52:52,894 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:52:52,987 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-16) [15c45473] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:52:53,014 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-16) [15c45473] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 19dd30c9 >2012-09-02 15:52:53,014 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [15c45473] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:52:53,014 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-16) [15c45473] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 19dd30c9 >2012-09-02 15:53:00,223 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp-/127.0.0.1:8009-7) Lock Acquired to object EngineLock [exclusiveLocks= key: 56a64ce1-90ab-4576-86fa-c269160aaa38 value: STORAGE >, sharedLocks= ] >2012-09-02 15:53:00,327 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (pool-4-thread-47) [277f3fd9] Running command: ActivateStorageDomainCommand internal: false. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 15:53:00,343 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (pool-4-thread-47) [277f3fd9] Lock freed to object EngineLock [exclusiveLocks= key: 56a64ce1-90ab-4576-86fa-c269160aaa38 value: STORAGE >, sharedLocks= ] >2012-09-02 15:53:00,348 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (pool-4-thread-47) [277f3fd9] ActivateStorage Domain. Before Connect all hosts to pool. Time:9/2/12 3:53 PM >2012-09-02 15:53:00,367 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-4-thread-48) [277f3fd9] START, ConnectStorageServerVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, storageType = ISCSI, connectionList = [{ id: 90cee3d3-f602-49f9-8f17-8665c22f2d06, connection: 10.35.64.10 };]), log id: 530b8a2f >2012-09-02 15:53:00,497 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-4-thread-48) [277f3fd9] FINISH, ConnectStorageServerVDSCommand, return: {90cee3d3-f602-49f9-8f17-8665c22f2d06=0}, log id: 530b8a2f >2012-09-02 15:53:00,497 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (pool-4-thread-47) [277f3fd9] ActivateStorage Domain. After Connect all hosts to pool. Time:9/2/12 3:53 PM >2012-09-02 15:53:00,498 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (pool-4-thread-47) [277f3fd9] START, ActivateStorageDomainVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, storageDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38), log id: 29b09089 >2012-09-02 15:53:00,523 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-4-thread-47) [277f3fd9] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:53:00,554 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-47) [277f3fd9] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 609a8ae6 >2012-09-02 15:53:01,491 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-4-thread-47) [277f3fd9] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 15:53:01,491 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-4-thread-47) [277f3fd9] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:53:01,491 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-4-thread-47) [277f3fd9] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:53:01,491 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-47) [277f3fd9] FINISH, ConnectStoragePoolVDSCommand, log id: 609a8ae6 >2012-09-02 15:53:01,491 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-4-thread-47) [277f3fd9] IrsBroker::Failed::ActivateStorageDomainVDS >2012-09-02 15:53:01,491 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-4-thread-47) [277f3fd9] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:53:01,557 WARN [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-4-thread-47) [456dba6d] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_STORAGE_DOMAIN_STATUS_ILLEGAL2,$status Locked >2012-09-02 15:53:01,565 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (pool-4-thread-47) [456dba6d] FINISH, ActivateStorageDomainVDSCommand, log id: 29b09089 >2012-09-02 15:53:01,565 ERROR [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (pool-4-thread-47) [456dba6d] Command org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.irsbroker.IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 15:53:01,578 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (pool-4-thread-47) [456dba6d] Command [id=9a38359a-5b51-48e6-907b-01fc21bf5bce]: Compensating CHANGED_STATUS_ONLY of org.ovirt.engine.core.common.businessentities.storage_pool_iso_map; snapshot: EntityStatusSnapshot [id=storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, storageId = 56a64ce1-90ab-4576-86fa-c269160aaa38, status=InActive]. >2012-09-02 15:53:03,052 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-22) [3b584081] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 15:53:03,078 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-22) [3b584081] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 38297e44 >2012-09-02 15:53:05,618 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-22) [3b584081] FINISH, ConnectStoragePoolVDSCommand, log id: 38297e44 >2012-09-02 15:53:05,815 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-22) [3b584081] hostFromVds::Connected host to pool - selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Free, storage pool TestDataCenter >2012-09-02 15:53:06,012 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-22) [3b584081] SpmStatus on vds b32eacf0-f4e9-11e1-973c-001a4a169705: Free >2012-09-02 15:53:06,029 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-22) [3b584081] starting spm on vds green-vdsa.qa.lab.tlv.redhat.com, storage pool TestDataCenter, prevId 1, LVER 0 >2012-09-02 15:53:06,030 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-22) [3b584081] START, SpmStartVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6d6a4413 >2012-09-02 15:53:06,053 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-22) [3b584081] spmStart polling started: taskId = 1292b047-4051-4587-a281-746de33b5d3d >2012-09-02 15:53:27,490 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-22) [3b584081] spmStart polling ended: taskId = 1292b047-4051-4587-a281-746de33b5d3d task status = finished >2012-09-02 15:53:27,509 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-22) [3b584081] spmStart polling ended. spm status: SPM >2012-09-02 15:53:27,510 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-22) [3b584081] START, HSMClearTaskVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, taskId=1292b047-4051-4587-a281-746de33b5d3d), log id: 2ec3da3b >2012-09-02 15:53:27,528 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-22) [3b584081] FINISH, HSMClearTaskVDSCommand, log id: 2ec3da3b >2012-09-02 15:53:27,529 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-22) [3b584081] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3c2c6f79, log id: 6d6a4413 >2012-09-02 15:53:27,542 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-22) [3b584081] Initialize Irs proxy from vds: 10.35.102.10 >2012-09-02 15:55:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-3) Autorecovering hosts is disabled, skipping >2012-09-02 15:55:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-3) Autorecovering storage domains is disabled, skipping >2012-09-02 15:57:37,015 INFO [org.ovirt.engine.core.bll.MaintananceNumberOfVdssCommand] (pool-4-thread-48) [388af135] Running command: MaintananceNumberOfVdssCommand internal: false. Entities affected : ID: 3880c572-f4f0-11e1-9a52-001a4a169705 Type: VDS >2012-09-02 15:57:37,015 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-48) [388af135] START, SetVdsStatusVDSCommand(vdsId = 3880c572-f4f0-11e1-9a52-001a4a169705, status=PreparingForMaintenance, nonOperationalReason=NONE), log id: 43715a6e >2012-09-02 15:57:37,052 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-48) [388af135] FINISH, SetVdsStatusVDSCommand, log id: 43715a6e >2012-09-02 15:57:37,141 INFO [org.ovirt.engine.core.bll.MaintananceVdsCommand] (pool-4-thread-48) [388af135] Running command: MaintananceVdsCommand internal: true. Entities affected : ID: 3880c572-f4f0-11e1-9a52-001a4a169705 Type: VDS >2012-09-02 15:57:37,636 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-25) vds::Updated vds status from Preparing for Maintenance to Maintenance in database, vds = 3880c572-f4f0-11e1-9a52-001a4a169705 : camel-vdsb.qa.lab.tlv.redhat.com >2012-09-02 15:57:37,667 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) Clearing cache of pool: c7387742-f4dc-11e1-8ca7-001a4a169705 for problematic entities of VDS: camel-vdsb.qa.lab.tlv.redhat.com. >2012-09-02 15:57:52,643 INFO [org.ovirt.engine.core.bll.ChangeVDSClusterCommand] (ajp-/127.0.0.1:8009-8) [6c81cf6c] Running command: ChangeVDSClusterCommand internal: false. Entities affected : ID: 3880c572-f4f0-11e1-9a52-001a4a169705 Type: VDS, ID: b1600626-f4e9-11e1-b9cc-001a4a169705 Type: VdsGroups >2012-09-02 15:57:52,677 INFO [org.ovirt.engine.core.bll.RemoveVdsSpmIdCommand] (ajp-/127.0.0.1:8009-8) [6c81cf6c] Running command: RemoveVdsSpmIdCommand internal: true. Entities affected : ID: 3880c572-f4f0-11e1-9a52-001a4a169705 Type: VDS >2012-09-02 15:57:52,705 INFO [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp-/127.0.0.1:8009-8) [6c81cf6c] Running command: AddVdsSpmIdCommand internal: true. Entities affected : ID: 3880c572-f4f0-11e1-9a52-001a4a169705 Type: VDS >2012-09-02 15:57:52,732 INFO [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp-/127.0.0.1:8009-8) [6c81cf6c] Lock freed to object EngineLock [exclusiveLocks= key: bd560b80-c245-46b3-ad8c-b142a3460cf6 value: REGISTER_VDS >, sharedLocks= ] >2012-09-02 15:57:52,794 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp-/127.0.0.1:8009-9) [5718743b] Running command: UpdateVdsCommand internal: false. Entities affected : ID: 3880c572-f4f0-11e1-9a52-001a4a169705 Type: VDS >2012-09-02 15:57:54,163 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (ajp-/127.0.0.1:8009-2) Lock Acquired to object EngineLock [exclusiveLocks= key: 3880c572-f4f0-11e1-9a52-001a4a169705 value: VDS >, sharedLocks= ] >2012-09-02 15:57:55,221 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (pool-4-thread-50) [2171672c] Running command: ActivateVdsCommand internal: false. Entities affected : ID: 3880c572-f4f0-11e1-9a52-001a4a169705 Type: VDS >2012-09-02 15:57:55,243 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-50) [2171672c] START, SetVdsStatusVDSCommand(vdsId = 3880c572-f4f0-11e1-9a52-001a4a169705, status=Unassigned, nonOperationalReason=NONE), log id: 221b97a8 >2012-09-02 15:57:55,245 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-50) [2171672c] FINISH, SetVdsStatusVDSCommand, log id: 221b97a8 >2012-09-02 15:57:55,256 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (pool-4-thread-50) [2171672c] START, ActivateVdsVDSCommand(vdsId = 3880c572-f4f0-11e1-9a52-001a4a169705), log id: 5c206674 >2012-09-02 15:57:55,541 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (pool-4-thread-50) [29618a24] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 3880c572-f4f0-11e1-9a52-001a4a169705 Type: VDS >2012-09-02 15:57:55,543 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (pool-4-thread-50) [5a3cd19e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 3880c572-f4f0-11e1-9a52-001a4a169705 Type: VDS >2012-09-02 15:57:55,544 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (pool-4-thread-50) [5a3cd19e] FINISH, ActivateVdsVDSCommand, log id: 5c206674 >2012-09-02 15:57:55,576 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (pool-4-thread-50) [5a3cd19e] Lock freed to object EngineLock [exclusiveLocks= key: 3880c572-f4f0-11e1-9a52-001a4a169705 value: VDS >, sharedLocks= ] >2012-09-02 15:57:55,957 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-48) [399e2839] Running command: InitVdsOnUpCommand internal: true. >2012-09-02 15:57:56,024 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (QuartzScheduler_Worker-48) [6b7ddd30] START, ValidateStorageServerConnectionVDSCommand(vdsId = 3880c572-f4f0-11e1-9a52-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, storageType = ISCSI, connectionList = [{ id: 90cee3d3-f602-49f9-8f17-8665c22f2d06, connection: 10.35.64.10 };]), log id: 54d544d2 >2012-09-02 15:57:56,047 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (QuartzScheduler_Worker-48) [6b7ddd30] FINISH, ValidateStorageServerConnectionVDSCommand, return: {90cee3d3-f602-49f9-8f17-8665c22f2d06=0}, log id: 54d544d2 >2012-09-02 15:57:56,047 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-48) [6b7ddd30] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: bd560b80-c245-46b3-ad8c-b142a3460cf6 Type: StoragePool >2012-09-02 15:57:56,048 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (QuartzScheduler_Worker-48) [6b7ddd30] START, ConnectStorageServerVDSCommand(vdsId = 3880c572-f4f0-11e1-9a52-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, storageType = ISCSI, connectionList = [{ id: 90cee3d3-f602-49f9-8f17-8665c22f2d06, connection: 10.35.64.10 };]), log id: 3c122ad2 >2012-09-02 15:57:56,157 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (QuartzScheduler_Worker-48) [6b7ddd30] FINISH, ConnectStorageServerVDSCommand, return: {90cee3d3-f602-49f9-8f17-8665c22f2d06=0}, log id: 3c122ad2 >2012-09-02 15:57:56,157 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-48) [6b7ddd30] Host camel-vdsb.qa.lab.tlv.redhat.com storage connection was succeeded >2012-09-02 15:57:56,179 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-48) [6b7ddd30] START, ConnectStoragePoolVDSCommand(vdsId = 3880c572-f4f0-11e1-9a52-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 2, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 712dfcf2 >2012-09-02 15:57:58,676 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-48) [6b7ddd30] FINISH, ConnectStoragePoolVDSCommand, log id: 712dfcf2 >2012-09-02 15:57:58,682 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-48) [6b7ddd30] No string for UNASSIGNED type. Use default Log >2012-09-02 15:57:58,683 INFO [org.ovirt.engine.core.bll.MultipleActionsRunner] (pool-4-thread-48) [6b7ddd30] MultipleActionsRunner of type MigrateVmToServer invoked with no actions >2012-09-02 15:57:58,698 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-48) [146d0172] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 3880c572-f4f0-11e1-9a52-001a4a169705 Type: VDS >2012-09-02 15:57:58,709 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-48) [1ea6b2c6] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 3880c572-f4f0-11e1-9a52-001a4a169705 Type: VDS >2012-09-02 15:58:15,105 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8009-3) Checking if user admin@internal is an admin, result true >2012-09-02 15:58:15,105 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8009-3) Running command: LoginUserCommand internal: false. >2012-09-02 15:58:15,163 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8009-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2012-09-02 15:58:15,665 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8009-1) Checking if user admin@internal is an admin, result true >2012-09-02 15:58:15,665 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8009-1) Running command: LoginUserCommand internal: false. >2012-09-02 15:58:15,722 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8009-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2012-09-02 15:58:53,888 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) [2adea929] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:59:33,849 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-84) [17f32511] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 15:59:34,658 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-81) [6977b3eb] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: bd560b80-c245-46b3-ad8c-b142a3460cf6 Type: StoragePool >2012-09-02 15:59:34,689 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-81) [6977b3eb] IrsBroker::Failed::GetStoragePoolInfoVDS due to: ConnectException: Connection refused >2012-09-02 15:59:34,693 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-81) [6977b3eb] Irs placed on server null failed. Proceed Failover >2012-09-02 15:59:34,716 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-81) [6977b3eb] hostFromVds::selectedVds - camel-vdsb.qa.lab.tlv.redhat.com, spmStatus Free, storage pool TestDataCenter >2012-09-02 15:59:34,738 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-81) [6977b3eb] SPM Init: could not find reported vds or not up - pool:TestDataCenter vds_spm_id: 1 >2012-09-02 15:59:34,740 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-81) [6977b3eb] SPM selection - vds seems as spm green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:59:34,740 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-81) [6977b3eb] START, SpmStopVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6), log id: 211ab4b >2012-09-02 15:59:34,741 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-81) [6977b3eb] SpmStopVDSCommand:: vds green-vdsa.qa.lab.tlv.redhat.com is in Connecting status - not performing spm stop, pool id bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:59:34,741 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-81) [6977b3eb] FINISH, SpmStopVDSCommand, log id: 211ab4b >2012-09-02 15:59:34,741 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-81) [6977b3eb] spm stop on spm failed, stopping spm selection! >2012-09-02 15:59:35,856 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-77) [71797221] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 15:59:37,863 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-89) [5fb3a86f] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 15:59:39,870 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-92) ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 15:59:41,877 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-91) [303ed861] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 15:59:43,884 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-87) [62351dda] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 15:59:44,766 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [7952176] hostFromVds::selectedVds - camel-vdsb.qa.lab.tlv.redhat.com, spmStatus Free, storage pool TestDataCenter >2012-09-02 15:59:44,787 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [7952176] SPM Init: could not find reported vds or not up - pool:TestDataCenter vds_spm_id: 1 >2012-09-02 15:59:44,788 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [7952176] SPM selection - vds seems as spm green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:59:44,789 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-98) [7952176] START, SpmStopVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6), log id: 4efb0d67 >2012-09-02 15:59:44,789 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-98) [7952176] SpmStopVDSCommand:: vds green-vdsa.qa.lab.tlv.redhat.com is in Connecting status - not performing spm stop, pool id bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:59:44,789 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-98) [7952176] FINISH, SpmStopVDSCommand, log id: 4efb0d67 >2012-09-02 15:59:44,789 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [7952176] spm stop on spm failed, stopping spm selection! >2012-09-02 15:59:45,891 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-78) [1081755f] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 15:59:47,898 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-7) ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 15:59:49,907 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-99) [d9b82ea] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 15:59:51,915 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-10) [2b98e94d] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 15:59:53,924 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-6) ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 15:59:54,829 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-11) [13caede9] hostFromVds::selectedVds - camel-vdsb.qa.lab.tlv.redhat.com, spmStatus Free, storage pool TestDataCenter >2012-09-02 15:59:54,832 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-11) [13caede9] SPM Init: could not find reported vds or not up - pool:TestDataCenter vds_spm_id: 1 >2012-09-02 15:59:54,833 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-11) [13caede9] SPM selection - vds seems as spm green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 15:59:54,835 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-11) [13caede9] START, SpmStopVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6), log id: 57bfb750 >2012-09-02 15:59:54,835 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-11) [13caede9] SpmStopVDSCommand:: vds green-vdsa.qa.lab.tlv.redhat.com is in Connecting status - not performing spm stop, pool id bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 15:59:54,835 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-11) [13caede9] FINISH, SpmStopVDSCommand, log id: 57bfb750 >2012-09-02 15:59:54,835 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-11) [13caede9] spm stop on spm failed, stopping spm selection! >2012-09-02 15:59:55,931 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-15) [5147a4f8] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 15:59:57,937 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-12) [5064a4c0] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 15:59:59,948 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-14) [2ff7a817] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 16:00:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-19) [229837ce] Autorecovering hosts is disabled, skipping >2012-09-02 16:00:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-19) [229837ce] Autorecovering storage domains is disabled, skipping >2012-09-02 16:00:01,954 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-24) [392574c] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 16:00:03,960 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-21) [2adea929] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 16:00:04,860 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-29) hostFromVds::selectedVds - camel-vdsb.qa.lab.tlv.redhat.com, spmStatus Free, storage pool TestDataCenter >2012-09-02 16:00:04,862 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-29) SPM Init: could not find reported vds or not up - pool:TestDataCenter vds_spm_id: 1 >2012-09-02 16:00:04,863 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-29) SPM selection - vds seems as spm green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 16:00:04,864 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-29) START, SpmStopVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6), log id: 5b31f805 >2012-09-02 16:00:04,864 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-29) SpmStopVDSCommand:: vds green-vdsa.qa.lab.tlv.redhat.com is in Connecting status - not performing spm stop, pool id bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 16:00:04,864 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-29) FINISH, SpmStopVDSCommand, log id: 5b31f805 >2012-09-02 16:00:04,864 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-29) spm stop on spm failed, stopping spm selection! >2012-09-02 16:00:05,967 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-31) [227b1500] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 16:00:07,975 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-32) [65b08d21] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 16:00:09,986 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-28) [164d9fbc] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 16:00:11,995 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-35) [6514e28e] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 16:00:14,002 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-39) [4fc83cbd] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 16:00:14,888 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [1cf6d845] hostFromVds::selectedVds - camel-vdsb.qa.lab.tlv.redhat.com, spmStatus Free, storage pool TestDataCenter >2012-09-02 16:00:14,890 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [1cf6d845] SPM Init: could not find reported vds or not up - pool:TestDataCenter vds_spm_id: 1 >2012-09-02 16:00:14,891 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [1cf6d845] SPM selection - vds seems as spm green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 16:00:14,892 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-42) [1cf6d845] START, SpmStopVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6), log id: 677a5379 >2012-09-02 16:00:14,892 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-42) [1cf6d845] SpmStopVDSCommand:: vds green-vdsa.qa.lab.tlv.redhat.com is in Connecting status - not performing spm stop, pool id bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 16:00:14,892 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-42) [1cf6d845] FINISH, SpmStopVDSCommand, log id: 677a5379 >2012-09-02 16:00:14,892 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [1cf6d845] spm stop on spm failed, stopping spm selection! >2012-09-02 16:00:16,009 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-47) [7061330f] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 16:00:18,016 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-45) [4a22f461] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 16:00:20,024 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-49) [767e356a] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 16:00:22,031 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-44) [30ec8757] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 16:00:24,039 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-50) [6dc69ea2] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 16:00:24,918 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) [dbe32c9] hostFromVds::selectedVds - camel-vdsb.qa.lab.tlv.redhat.com, spmStatus Free, storage pool TestDataCenter >2012-09-02 16:00:24,920 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) [dbe32c9] SPM Init: could not find reported vds or not up - pool:TestDataCenter vds_spm_id: 1 >2012-09-02 16:00:24,921 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) [dbe32c9] SPM selection - vds seems as spm green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 16:00:24,922 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-60) [dbe32c9] START, SpmStopVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6), log id: 38b910c1 >2012-09-02 16:00:24,922 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-60) [dbe32c9] SpmStopVDSCommand:: vds green-vdsa.qa.lab.tlv.redhat.com is in Connecting status - not performing spm stop, pool id bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 16:00:24,922 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-60) [dbe32c9] FINISH, SpmStopVDSCommand, log id: 38b910c1 >2012-09-02 16:00:24,922 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) [dbe32c9] spm stop on spm failed, stopping spm selection! >2012-09-02 16:00:26,046 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-58) ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 16:00:28,068 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-62) [3ea43f70] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@1ce7e715 >2012-09-02 16:00:28,068 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-62) [3ea43f70] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 16:00:28,069 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-62) [3ea43f70] Command GetCapabilitiesVDS execution failed. Error: VDSRecoveringException: Recovering from crash or Initializing >2012-09-02 16:00:28,092 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-62) [3ea43f70] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, error = Recovering from crash or Initializing, continuing. >2012-09-02 16:00:30,897 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-55) [2e050512] Cancelling the recovery from crash timer for VDS b32eacf0-f4e9-11e1-973c-001a4a169705 because vds started initializing >2012-09-02 16:00:30,898 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-55) [49f6fa50] Running command: InitVdsOnUpCommand internal: true. >2012-09-02 16:00:30,958 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (QuartzScheduler_Worker-55) [3d10800c] START, ValidateStorageServerConnectionVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, storageType = ISCSI, connectionList = [{ id: 90cee3d3-f602-49f9-8f17-8665c22f2d06, connection: 10.35.64.10 };]), log id: 1c6a7e48 >2012-09-02 16:00:30,978 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (QuartzScheduler_Worker-55) [3d10800c] FINISH, ValidateStorageServerConnectionVDSCommand, return: {90cee3d3-f602-49f9-8f17-8665c22f2d06=0}, log id: 1c6a7e48 >2012-09-02 16:00:30,978 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-55) [3d10800c] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: bd560b80-c245-46b3-ad8c-b142a3460cf6 Type: StoragePool >2012-09-02 16:00:30,979 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (QuartzScheduler_Worker-55) [3d10800c] START, ConnectStorageServerVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, storageType = ISCSI, connectionList = [{ id: 90cee3d3-f602-49f9-8f17-8665c22f2d06, connection: 10.35.64.10 };]), log id: 34a4b68a >2012-09-02 16:00:31,089 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (QuartzScheduler_Worker-55) [3d10800c] FINISH, ConnectStorageServerVDSCommand, return: {90cee3d3-f602-49f9-8f17-8665c22f2d06=0}, log id: 34a4b68a >2012-09-02 16:00:31,089 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-55) [3d10800c] Host green-vdsa.qa.lab.tlv.redhat.com storage connection was succeeded >2012-09-02 16:00:31,109 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-55) [3d10800c] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 49d603a3 >2012-09-02 16:00:32,008 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-55) [3d10800c] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 16:00:32,008 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-55) [3d10800c] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 16:00:32,008 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-55) [3d10800c] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 16:00:32,008 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-55) [3d10800c] FINISH, ConnectStoragePoolVDSCommand, log id: 49d603a3 >2012-09-02 16:00:32,008 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-55) [3d10800c] Could not connect host green-vdsa.qa.lab.tlv.redhat.com to pool TestDataCenter >2012-09-02 16:00:32,010 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-55) [188e834] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: b32eacf0-f4e9-11e1-973c-001a4a169705 Type: VDS >2012-09-02 16:00:32,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-55) [188e834] START, SetVdsStatusVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 7d673fd1 >2012-09-02 16:00:32,013 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-55) [188e834] SetVdsStatusVDSCommand::VSD green-vdsa.qa.lab.tlv.redhat.com is spm and moved from up calling ResetIrs. >2012-09-02 16:00:32,013 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-55) [188e834] START, ResetIrsVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = 10.35.102.10, vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, ignoreStopFailed = false), log id: 78154855 >2012-09-02 16:00:32,014 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-55) [188e834] START, SpmStopVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6), log id: 5dc270e5 >2012-09-02 16:00:32,034 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-55) [188e834] SpmStopVDSCommand::Stopping SPM on vds green-vdsa.qa.lab.tlv.redhat.com, pool id bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 16:00:32,058 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-55) [188e834] FINISH, SpmStopVDSCommand, log id: 5dc270e5 >2012-09-02 16:00:32,059 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-55) [188e834] FINISH, ResetIrsVDSCommand, log id: 78154855 >2012-09-02 16:00:32,059 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-55) [188e834] FINISH, SetVdsStatusVDSCommand, log id: 7d673fd1 >2012-09-02 16:00:32,061 ERROR [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (pool-4-thread-50) [188e834] Failed to create ExecutionContext for MigrateVmCommand: java.lang.NullPointerException > at org.ovirt.engine.core.bll.MaintananceVdsCommand.createMigrateVmContext(MaintananceVdsCommand.java:114) [engine-bll.jar:] > at org.ovirt.engine.core.bll.MaintananceVdsCommand.MigrateAllVms(MaintananceVdsCommand.java:93) [engine-bll.jar:] > at org.ovirt.engine.core.bll.MaintananceVdsCommand.MigrateAllVms(MaintananceVdsCommand.java:79) [engine-bll.jar:] > at org.ovirt.engine.core.bll.SetNonOperationalVdsCommand$1.run(SetNonOperationalVdsCommand.java:44) [engine-bll.jar:] > at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalWrapperRunnable.run(ThreadPoolUtil.java:64) [engine-utils.jar:] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_05-icedtea] > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) [rt.jar:1.7.0_05-icedtea] > at java.util.concurrent.FutureTask.run(FutureTask.java:166) [rt.jar:1.7.0_05-icedtea] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [rt.jar:1.7.0_05-icedtea] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [rt.jar:1.7.0_05-icedtea] > at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_05-icedtea] > >2012-09-02 16:00:32,104 INFO [org.ovirt.engine.core.bll.MultipleActionsRunner] (pool-4-thread-48) [188e834] MultipleActionsRunner of type MigrateVmToServer invoked with no actions >2012-09-02 16:00:32,114 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-55) [64b6c5fb] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: b32eacf0-f4e9-11e1-973c-001a4a169705 Type: VDS >2012-09-02 16:00:32,123 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-55) [1bbf829c] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: b32eacf0-f4e9-11e1-973c-001a4a169705 Type: VDS >2012-09-02 16:00:32,124 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-55) [1bbf829c] Host b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2012-09-02 16:00:32,132 INFO [org.ovirt.engine.core.bll.InternalMigrateVmCommand] (pool-4-thread-50) [601be6cb] Running command: InternalMigrateVmCommand internal: true. Entities affected : ID: 5449d6cf-0e31-44b5-b0a9-65036f122662 Type: VM >2012-09-02 16:00:32,142 INFO [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand] (pool-4-thread-50) [601be6cb] START, MigrateVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, vmId=5449d6cf-0e31-44b5-b0a9-65036f122662, srcHost=10.35.102.10, dstVdsId=3880c572-f4f0-11e1-9a52-001a4a169705, dstHost=10.35.116.2:54321, migrationMethod=ONLINE), log id: 6a15633f >2012-09-02 16:00:32,150 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] (pool-4-thread-50) [601be6cb] VdsBroker::migrate::Entered (vm_guid=5449d6cf-0e31-44b5-b0a9-65036f122662, srcHost=10.35.102.10, dstHost=10.35.116.2:54321, method=online >2012-09-02 16:00:32,150 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] (pool-4-thread-50) [601be6cb] START, MigrateBrokerVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, vmId=5449d6cf-0e31-44b5-b0a9-65036f122662, srcHost=10.35.102.10, dstVdsId=3880c572-f4f0-11e1-9a52-001a4a169705, dstHost=10.35.116.2:54321, migrationMethod=ONLINE), log id: 168a5a0e >2012-09-02 16:00:32,182 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] (pool-4-thread-50) [601be6cb] FINISH, MigrateBrokerVDSCommand, log id: 168a5a0e >2012-09-02 16:00:32,197 INFO [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand] (pool-4-thread-50) [601be6cb] FINISH, MigrateVDSCommand, return: MigratingFrom, log id: 6a15633f >2012-09-02 16:00:34,600 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-66) [60860e0e] vds::refreshVmList vm id 5449d6cf-0e31-44b5-b0a9-65036f122662 is migrating to vds camel-vdsb.qa.lab.tlv.redhat.com ignoring it in the refresh till migration is done >2012-09-02 16:00:34,948 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [2ab07b12] hostFromVds::selectedVds - camel-vdsb.qa.lab.tlv.redhat.com, spmStatus Free, storage pool TestDataCenter >2012-09-02 16:00:34,992 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [2ab07b12] SpmStatus on vds b32eacf0-f4e9-11e1-973c-001a4a169705: Unknown_Pool >2012-09-02 16:00:35,032 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [2ab07b12] starting spm on vds camel-vdsb.qa.lab.tlv.redhat.com, storage pool TestDataCenter, prevId 1, LVER 1 >2012-09-02 16:00:35,033 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [2ab07b12] START, SpmStartVDSCommand(vdsId = 3880c572-f4f0-11e1-9a52-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, prevId=1, prevLVER=1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5aea56ec >2012-09-02 16:00:35,061 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [2ab07b12] spmStart polling started: taskId = 47084d0c-3997-4ffd-9205-664eded4de09 >2012-09-02 16:00:36,187 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (QuartzScheduler_Worker-57) [6af76567] START, DestroyVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, vmId=5449d6cf-0e31-44b5-b0a9-65036f122662, force=false, secondsToWait=0, gracefully=false), log id: 671d2e4d >2012-09-02 16:00:36,256 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (QuartzScheduler_Worker-57) [6af76567] FINISH, DestroyVDSCommand, log id: 671d2e4d >2012-09-02 16:00:36,257 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-57) [6af76567] Setting VM testVm 5449d6cf-0e31-44b5-b0a9-65036f122662 to status unknown >2012-09-02 16:00:36,257 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-57) [6af76567] vm testVm running in db and not running in vds - add to rerun treatment. vds green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 16:00:36,342 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (QuartzScheduler_Worker-69) [39fd24f3] START, FullListVdsCommand(vdsId = 3880c572-f4f0-11e1-9a52-001a4a169705, vds=null, vmIds=[5449d6cf-0e31-44b5-b0a9-65036f122662]), log id: 2bfd395f >2012-09-02 16:00:36,372 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (QuartzScheduler_Worker-69) [39fd24f3] FINISH, FullListVdsCommand, return: [Lorg.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcStruct;@3395cf38, log id: 2bfd395f >2012-09-02 16:00:42,241 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [2ab07b12] spmStart polling ended: taskId = 47084d0c-3997-4ffd-9205-664eded4de09 task status = finished >2012-09-02 16:00:42,261 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [2ab07b12] spmStart polling ended. spm status: SPM >2012-09-02 16:00:42,262 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-72) [2ab07b12] START, HSMClearTaskVDSCommand(vdsId = 3880c572-f4f0-11e1-9a52-001a4a169705, taskId=47084d0c-3997-4ffd-9205-664eded4de09), log id: ac51a9 >2012-09-02 16:00:42,282 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-72) [2ab07b12] FINISH, HSMClearTaskVDSCommand, log id: ac51a9 >2012-09-02 16:00:42,282 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [2ab07b12] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@31252e97, log id: 5aea56ec >2012-09-02 16:00:42,290 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [2ab07b12] Initialize Irs proxy from vds: 10.35.116.2 >2012-09-02 16:00:42,344 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-84) [17f32511] Attempting to get and stop tasks on storage pool TestDataCenter >2012-09-02 16:00:42,345 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-84) [17f32511] START, SPMGetAllTasksInfoVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6610497e >2012-09-02 16:00:42,397 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-84) [17f32511] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 16:00:42,399 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-84) [17f32511] START, HSMGetAllTasksInfoVDSCommand(vdsId = 3880c572-f4f0-11e1-9a52-001a4a169705), log id: 9a38836 >2012-09-02 16:00:42,420 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-84) [17f32511] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 9a38836 >2012-09-02 16:00:42,420 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-84) [17f32511] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 6610497e >2012-09-02 16:00:42,420 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-84) [17f32511] Discovered no tasks on Storage Pool TestDataCenter >2012-09-02 16:00:42,421 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-84) [17f32511] START, SPMGetAllTasksInfoVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 604f3dcd >2012-09-02 16:00:42,421 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-84) [17f32511] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 16:00:42,422 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-84) [17f32511] START, HSMGetAllTasksInfoVDSCommand(vdsId = 3880c572-f4f0-11e1-9a52-001a4a169705), log id: 5402e0d7 >2012-09-02 16:00:42,442 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-84) [17f32511] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 5402e0d7 >2012-09-02 16:00:42,442 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-84) [17f32511] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 604f3dcd >2012-09-02 16:00:42,442 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-84) [17f32511] Discovered no tasks on Storage Pool TestDataCenter >2012-09-02 16:02:08,900 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (ajp-/127.0.0.1:8009-4) Lock Acquired to object EngineLock [exclusiveLocks= key: b32eacf0-f4e9-11e1-973c-001a4a169705 value: VDS >, sharedLocks= ] >2012-09-02 16:02:08,949 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (pool-4-thread-47) [78f84b1] Running command: ActivateVdsCommand internal: false. Entities affected : ID: b32eacf0-f4e9-11e1-973c-001a4a169705 Type: VDS >2012-09-02 16:02:08,951 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-47) [78f84b1] START, SetVdsStatusVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, status=Unassigned, nonOperationalReason=NONE), log id: 3f778049 >2012-09-02 16:02:08,954 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-47) [78f84b1] FINISH, SetVdsStatusVDSCommand, log id: 3f778049 >2012-09-02 16:02:08,962 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (pool-4-thread-47) [78f84b1] START, ActivateVdsVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705), log id: 25a1662e >2012-09-02 16:02:09,335 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (pool-4-thread-47) [3162a8ff] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: b32eacf0-f4e9-11e1-973c-001a4a169705 Type: VDS >2012-09-02 16:02:09,337 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (pool-4-thread-47) [5fd5d1e1] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: b32eacf0-f4e9-11e1-973c-001a4a169705 Type: VDS >2012-09-02 16:02:09,338 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (pool-4-thread-47) [5fd5d1e1] FINISH, ActivateVdsVDSCommand, log id: 25a1662e >2012-09-02 16:02:09,369 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (pool-4-thread-47) [5fd5d1e1] Lock freed to object EngineLock [exclusiveLocks= key: b32eacf0-f4e9-11e1-973c-001a4a169705 value: VDS >, sharedLocks= ] >2012-09-02 16:02:10,839 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-10) [67005bb8] Running command: InitVdsOnUpCommand internal: true. >2012-09-02 16:02:10,903 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (QuartzScheduler_Worker-10) [51555107] START, ValidateStorageServerConnectionVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, storageType = ISCSI, connectionList = [{ id: 90cee3d3-f602-49f9-8f17-8665c22f2d06, connection: 10.35.64.10 };]), log id: 72177afe >2012-09-02 16:02:10,934 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (QuartzScheduler_Worker-10) [51555107] FINISH, ValidateStorageServerConnectionVDSCommand, return: {90cee3d3-f602-49f9-8f17-8665c22f2d06=0}, log id: 72177afe >2012-09-02 16:02:10,934 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-10) [51555107] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: bd560b80-c245-46b3-ad8c-b142a3460cf6 Type: StoragePool >2012-09-02 16:02:10,935 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (QuartzScheduler_Worker-10) [51555107] START, ConnectStorageServerVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, storageType = ISCSI, connectionList = [{ id: 90cee3d3-f602-49f9-8f17-8665c22f2d06, connection: 10.35.64.10 };]), log id: 7d13aeaf >2012-09-02 16:02:11,050 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (QuartzScheduler_Worker-10) [51555107] FINISH, ConnectStorageServerVDSCommand, return: {90cee3d3-f602-49f9-8f17-8665c22f2d06=0}, log id: 7d13aeaf >2012-09-02 16:02:11,050 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-10) [51555107] Host green-vdsa.qa.lab.tlv.redhat.com storage connection was succeeded >2012-09-02 16:02:11,065 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-10) [51555107] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 5c60e082 >2012-09-02 16:02:13,755 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-10) [51555107] FINISH, ConnectStoragePoolVDSCommand, log id: 5c60e082 >2012-09-02 16:02:13,760 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-10) [51555107] No string for UNASSIGNED type. Use default Log >2012-09-02 16:02:13,761 INFO [org.ovirt.engine.core.bll.MultipleActionsRunner] (pool-4-thread-50) [51555107] MultipleActionsRunner of type MigrateVmToServer invoked with no actions >2012-09-02 16:02:13,772 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-10) [4d95ee3f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: b32eacf0-f4e9-11e1-973c-001a4a169705 Type: VDS >2012-09-02 16:02:13,790 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-10) [29507acf] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: b32eacf0-f4e9-11e1-973c-001a4a169705 Type: VDS >2012-09-02 16:02:21,932 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-20) Domain 56a64ce1-90ab-4576-86fa-c269160aaa38 recovered from problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 16:02:21,932 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-20) Domain 56a64ce1-90ab-4576-86fa-c269160aaa38 has recovered from problem. No active host in the DC is reporting it as problematic, so clearing the domain recovery timer. >2012-09-02 16:03:41,969 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-35) [6514e28e] DbUserCacheManager::refreshAllUserData() - entered >2012-09-02 16:05:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-52) [570552a] Autorecovering hosts is disabled, skipping >2012-09-02 16:05:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-52) [570552a] Autorecovering storage domains is disabled, skipping >2012-09-02 16:05:41,011 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8009-5) Checking if user admin@internal is an admin, result true >2012-09-02 16:05:41,011 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8009-5) Running command: LoginUserCommand internal: false. >2012-09-02 16:05:41,072 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8009-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2012-09-02 16:05:41,267 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8009-3) Checking if user admin@internal is an admin, result true >2012-09-02 16:05:41,267 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8009-3) Running command: LoginUserCommand internal: false. >2012-09-02 16:05:41,314 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8009-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2012-09-02 16:06:09,136 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-56) domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 16:10:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-84) [17f32511] Autorecovering hosts is disabled, skipping >2012-09-02 16:10:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-84) [17f32511] Autorecovering storage domains is disabled, skipping >2012-09-02 16:11:09,136 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [89b785c] starting ProcessDomainRecovery for domain 56a64ce1-90ab-4576-86fa-c269160aaa38 >2012-09-02 16:11:09,628 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [89b785c] vds green-vdsa.qa.lab.tlv.redhat.com reported domain 56a64ce1-90ab-4576-86fa-c269160aaa38:ISCSIDataDomain as in problem, moving the vds to status NonOperational >2012-09-02 16:11:09,633 WARN [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (QuartzScheduler_Worker-71) [7a07ce1a] The message key SetNonOperationalVds is missing from bundles/ExecutionMessages >2012-09-02 16:11:09,980 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-71) [7a07ce1a] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: b32eacf0-f4e9-11e1-973c-001a4a169705 Type: VDS >2012-09-02 16:11:09,981 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-71) [7a07ce1a] START, SetVdsStatusVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 26021fee >2012-09-02 16:11:09,983 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-71) [7a07ce1a] FINISH, SetVdsStatusVDSCommand, log id: 26021fee >2012-09-02 16:11:10,221 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [7a07ce1a] Clearing cache of pool: bd560b80-c245-46b3-ad8c-b142a3460cf6 for problematic entities of VDS: green-vdsa.qa.lab.tlv.redhat.com. >2012-09-02 16:11:10,222 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [7a07ce1a] Domain 56a64ce1-90ab-4576-86fa-c269160aaa38 recovered from problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 16:11:10,222 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [7a07ce1a] Domain 56a64ce1-90ab-4576-86fa-c269160aaa38 has recovered from problem. No active host in the DC is reporting it as problematic, so clearing the domain recovery timer. >2012-09-02 16:11:16,715 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (ajp-/127.0.0.1:8009-8) [4da597ba] Lock Acquired to object EngineLock [exclusiveLocks= key: b32eacf0-f4e9-11e1-973c-001a4a169705 value: VDS >, sharedLocks= ] >2012-09-02 16:11:16,754 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (ajp-/127.0.0.1:8009-8) [4da597ba] Running command: ActivateVdsCommand internal: false. Entities affected : ID: b32eacf0-f4e9-11e1-973c-001a4a169705 Type: VDS >2012-09-02 16:11:16,756 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp-/127.0.0.1:8009-8) [4da597ba] START, SetVdsStatusVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, status=Unassigned, nonOperationalReason=NONE), log id: 60c3460a >2012-09-02 16:11:16,758 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp-/127.0.0.1:8009-8) [4da597ba] FINISH, SetVdsStatusVDSCommand, log id: 60c3460a >2012-09-02 16:11:16,768 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (ajp-/127.0.0.1:8009-8) [4da597ba] START, ActivateVdsVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705), log id: bc1135f >2012-09-02 16:11:17,530 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (ajp-/127.0.0.1:8009-8) [f0d51ef] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: b32eacf0-f4e9-11e1-973c-001a4a169705 Type: VDS >2012-09-02 16:11:17,532 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (ajp-/127.0.0.1:8009-8) [60770ca1] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: b32eacf0-f4e9-11e1-973c-001a4a169705 Type: VDS >2012-09-02 16:11:17,534 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (ajp-/127.0.0.1:8009-8) [60770ca1] FINISH, ActivateVdsVDSCommand, log id: bc1135f >2012-09-02 16:11:17,592 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (ajp-/127.0.0.1:8009-8) [60770ca1] Lock freed to object EngineLock [exclusiveLocks= key: b32eacf0-f4e9-11e1-973c-001a4a169705 value: VDS >, sharedLocks= ] >2012-09-02 16:11:17,873 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-93) [6afbd5aa] Running command: InitVdsOnUpCommand internal: true. >2012-09-02 16:11:17,954 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (QuartzScheduler_Worker-93) [b106054] START, ValidateStorageServerConnectionVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, storageType = ISCSI, connectionList = [{ id: 90cee3d3-f602-49f9-8f17-8665c22f2d06, connection: 10.35.64.10 };]), log id: 5bfdb474 >2012-09-02 16:11:17,974 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (QuartzScheduler_Worker-93) [b106054] FINISH, ValidateStorageServerConnectionVDSCommand, return: {90cee3d3-f602-49f9-8f17-8665c22f2d06=0}, log id: 5bfdb474 >2012-09-02 16:11:17,974 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-93) [b106054] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: bd560b80-c245-46b3-ad8c-b142a3460cf6 Type: StoragePool >2012-09-02 16:11:17,975 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (QuartzScheduler_Worker-93) [b106054] START, ConnectStorageServerVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, storageType = ISCSI, connectionList = [{ id: 90cee3d3-f602-49f9-8f17-8665c22f2d06, connection: 10.35.64.10 };]), log id: 1ac07ce7 >2012-09-02 16:11:18,095 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (QuartzScheduler_Worker-93) [b106054] FINISH, ConnectStorageServerVDSCommand, return: {90cee3d3-f602-49f9-8f17-8665c22f2d06=0}, log id: 1ac07ce7 >2012-09-02 16:11:18,096 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-93) [b106054] Host green-vdsa.qa.lab.tlv.redhat.com storage connection was succeeded >2012-09-02 16:11:18,114 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-93) [b106054] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 44a211f1 >2012-09-02 16:11:19,010 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-93) [b106054] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 16:11:19,011 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-93) [b106054] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 16:11:19,011 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-93) [b106054] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 16:11:19,011 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-93) [b106054] FINISH, ConnectStoragePoolVDSCommand, log id: 44a211f1 >2012-09-02 16:11:19,011 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-93) [b106054] Could not connect host green-vdsa.qa.lab.tlv.redhat.com to pool TestDataCenter >2012-09-02 16:11:19,013 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-93) [3c669fa0] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: b32eacf0-f4e9-11e1-973c-001a4a169705 Type: VDS >2012-09-02 16:11:19,013 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-93) [3c669fa0] START, SetVdsStatusVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 4418dde5 >2012-09-02 16:11:19,016 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-93) [3c669fa0] FINISH, SetVdsStatusVDSCommand, log id: 4418dde5 >2012-09-02 16:11:19,233 INFO [org.ovirt.engine.core.bll.MultipleActionsRunner] (pool-4-thread-47) [3c669fa0] MultipleActionsRunner of type MigrateVmToServer invoked with no actions >2012-09-02 16:11:19,294 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-93) [32cdf6e9] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: b32eacf0-f4e9-11e1-973c-001a4a169705 Type: VDS >2012-09-02 16:11:19,347 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-93) [43d82869] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: b32eacf0-f4e9-11e1-973c-001a4a169705 Type: VDS >2012-09-02 16:11:19,348 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-93) [43d82869] Host b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2012-09-02 16:15:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-13) Autorecovering hosts is disabled, skipping >2012-09-02 16:15:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-13) Autorecovering storage domains is disabled, skipping >2012-09-02 16:20:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-39) [4fc83cbd] Autorecovering hosts is disabled, skipping >2012-09-02 16:20:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-39) [4fc83cbd] Autorecovering storage domains is disabled, skipping >2012-09-02 16:22:20,335 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (ajp-/127.0.0.1:8009-1) Lock Acquired to object EngineLock [exclusiveLocks= key: b32eacf0-f4e9-11e1-973c-001a4a169705 value: VDS >, sharedLocks= ] >2012-09-02 16:22:20,377 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (pool-4-thread-47) [f7a6db7] Running command: ActivateVdsCommand internal: false. Entities affected : ID: b32eacf0-f4e9-11e1-973c-001a4a169705 Type: VDS >2012-09-02 16:22:20,379 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-47) [f7a6db7] START, SetVdsStatusVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, status=Unassigned, nonOperationalReason=NONE), log id: 2b57ee78 >2012-09-02 16:22:20,381 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-47) [f7a6db7] FINISH, SetVdsStatusVDSCommand, log id: 2b57ee78 >2012-09-02 16:22:20,392 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (pool-4-thread-47) [f7a6db7] START, ActivateVdsVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705), log id: 58772f46 >2012-09-02 16:22:20,720 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (pool-4-thread-47) [47fd9632] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: b32eacf0-f4e9-11e1-973c-001a4a169705 Type: VDS >2012-09-02 16:22:20,722 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (pool-4-thread-47) [56f0f600] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: b32eacf0-f4e9-11e1-973c-001a4a169705 Type: VDS >2012-09-02 16:22:20,724 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (pool-4-thread-47) [56f0f600] FINISH, ActivateVdsVDSCommand, log id: 58772f46 >2012-09-02 16:22:20,765 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (pool-4-thread-47) [56f0f600] Lock freed to object EngineLock [exclusiveLocks= key: b32eacf0-f4e9-11e1-973c-001a4a169705 value: VDS >, sharedLocks= ] >2012-09-02 16:22:21,170 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-39) [55358fc5] Running command: InitVdsOnUpCommand internal: true. >2012-09-02 16:22:23,078 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (QuartzScheduler_Worker-39) [1b744607] START, ValidateStorageServerConnectionVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, storageType = ISCSI, connectionList = [{ id: 90cee3d3-f602-49f9-8f17-8665c22f2d06, connection: 10.35.64.10 };]), log id: 5e573dc5 >2012-09-02 16:22:23,103 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (QuartzScheduler_Worker-39) [1b744607] FINISH, ValidateStorageServerConnectionVDSCommand, return: {90cee3d3-f602-49f9-8f17-8665c22f2d06=0}, log id: 5e573dc5 >2012-09-02 16:22:23,103 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-39) [1b744607] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: bd560b80-c245-46b3-ad8c-b142a3460cf6 Type: StoragePool >2012-09-02 16:22:23,104 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (QuartzScheduler_Worker-39) [1b744607] START, ConnectStorageServerVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, storageType = ISCSI, connectionList = [{ id: 90cee3d3-f602-49f9-8f17-8665c22f2d06, connection: 10.35.64.10 };]), log id: 3a96a6a1 >2012-09-02 16:22:23,224 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (QuartzScheduler_Worker-39) [1b744607] FINISH, ConnectStorageServerVDSCommand, return: {90cee3d3-f602-49f9-8f17-8665c22f2d06=0}, log id: 3a96a6a1 >2012-09-02 16:22:23,224 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-39) [1b744607] Host green-vdsa.qa.lab.tlv.redhat.com storage connection was succeeded >2012-09-02 16:22:23,255 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-39) [1b744607] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 4e8a9773 >2012-09-02 16:22:24,493 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-39) [1b744607] FINISH, ConnectStoragePoolVDSCommand, log id: 4e8a9773 >2012-09-02 16:22:24,497 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-39) [1b744607] No string for UNASSIGNED type. Use default Log >2012-09-02 16:22:24,499 INFO [org.ovirt.engine.core.bll.MultipleActionsRunner] (pool-4-thread-50) [1b744607] MultipleActionsRunner of type MigrateVmToServer invoked with no actions >2012-09-02 16:22:24,518 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-39) [696a9923] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: b32eacf0-f4e9-11e1-973c-001a4a169705 Type: VDS >2012-09-02 16:22:24,562 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-39) [62249710] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: b32eacf0-f4e9-11e1-973c-001a4a169705 Type: VDS >2012-09-02 16:22:26,655 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-45) [4a22f461] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 16:25:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-63) [6267f606] Autorecovering hosts is disabled, skipping >2012-09-02 16:25:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-63) [6267f606] Autorecovering storage domains is disabled, skipping >2012-09-02 16:27:26,656 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-70) starting ProcessDomainRecovery for domain 56a64ce1-90ab-4576-86fa-c269160aaa38 >2012-09-02 16:27:26,985 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-70) vds green-vdsa.qa.lab.tlv.redhat.com reported domain 56a64ce1-90ab-4576-86fa-c269160aaa38:ISCSIDataDomain as in problem, moving the vds to status NonOperational >2012-09-02 16:27:26,989 WARN [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (QuartzScheduler_Worker-70) [4679b9c1] The message key SetNonOperationalVds is missing from bundles/ExecutionMessages >2012-09-02 16:27:27,029 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-70) [4679b9c1] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: b32eacf0-f4e9-11e1-973c-001a4a169705 Type: VDS >2012-09-02 16:27:27,029 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-70) [4679b9c1] START, SetVdsStatusVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 60b169ee >2012-09-02 16:27:27,032 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-70) [4679b9c1] FINISH, SetVdsStatusVDSCommand, log id: 60b169ee >2012-09-02 16:27:27,115 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-70) [4679b9c1] Clearing cache of pool: bd560b80-c245-46b3-ad8c-b142a3460cf6 for problematic entities of VDS: green-vdsa.qa.lab.tlv.redhat.com. >2012-09-02 16:27:27,115 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-70) [4679b9c1] Domain 56a64ce1-90ab-4576-86fa-c269160aaa38 recovered from problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 16:27:27,115 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-70) [4679b9c1] Domain 56a64ce1-90ab-4576-86fa-c269160aaa38 has recovered from problem. No active host in the DC is reporting it as problematic, so clearing the domain recovery timer. >2012-09-02 16:30:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-95) [4fcef51] Autorecovering hosts is disabled, skipping >2012-09-02 16:30:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-95) [4fcef51] Autorecovering storage domains is disabled, skipping >2012-09-02 16:32:53,207 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (ajp-/127.0.0.1:8009-11) Lock Acquired to object EngineLock [exclusiveLocks= key: b32eacf0-f4e9-11e1-973c-001a4a169705 value: VDS >, sharedLocks= ] >2012-09-02 16:32:53,269 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (pool-4-thread-48) [5dc10b56] Running command: ActivateVdsCommand internal: false. Entities affected : ID: b32eacf0-f4e9-11e1-973c-001a4a169705 Type: VDS >2012-09-02 16:32:53,271 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-48) [5dc10b56] START, SetVdsStatusVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, status=Unassigned, nonOperationalReason=NONE), log id: 2b56e6c4 >2012-09-02 16:32:53,274 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-48) [5dc10b56] FINISH, SetVdsStatusVDSCommand, log id: 2b56e6c4 >2012-09-02 16:32:53,280 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (pool-4-thread-48) [5dc10b56] START, ActivateVdsVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705), log id: 70f775b5 >2012-09-02 16:32:53,988 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (pool-4-thread-48) [7e28df86] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: b32eacf0-f4e9-11e1-973c-001a4a169705 Type: VDS >2012-09-02 16:32:53,990 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (pool-4-thread-48) [76953e5d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: b32eacf0-f4e9-11e1-973c-001a4a169705 Type: VDS >2012-09-02 16:32:53,991 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (pool-4-thread-48) [76953e5d] FINISH, ActivateVdsVDSCommand, log id: 70f775b5 >2012-09-02 16:32:54,055 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (pool-4-thread-48) [76953e5d] Lock freed to object EngineLock [exclusiveLocks= key: b32eacf0-f4e9-11e1-973c-001a4a169705 value: VDS >, sharedLocks= ] >2012-09-02 16:32:54,349 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-38) [4aeb230d] Running command: InitVdsOnUpCommand internal: true. >2012-09-02 16:32:54,405 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (QuartzScheduler_Worker-38) [5cfe1c0b] START, ValidateStorageServerConnectionVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, storageType = ISCSI, connectionList = [{ id: 90cee3d3-f602-49f9-8f17-8665c22f2d06, connection: 10.35.64.10 };]), log id: 426b18a7 >2012-09-02 16:32:54,430 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (QuartzScheduler_Worker-38) [5cfe1c0b] FINISH, ValidateStorageServerConnectionVDSCommand, return: {90cee3d3-f602-49f9-8f17-8665c22f2d06=0}, log id: 426b18a7 >2012-09-02 16:32:54,430 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-38) [5cfe1c0b] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: bd560b80-c245-46b3-ad8c-b142a3460cf6 Type: StoragePool >2012-09-02 16:32:54,431 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (QuartzScheduler_Worker-38) [5cfe1c0b] START, ConnectStorageServerVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, storageType = ISCSI, connectionList = [{ id: 90cee3d3-f602-49f9-8f17-8665c22f2d06, connection: 10.35.64.10 };]), log id: 5a6c565d >2012-09-02 16:32:54,556 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (QuartzScheduler_Worker-38) [5cfe1c0b] FINISH, ConnectStorageServerVDSCommand, return: {90cee3d3-f602-49f9-8f17-8665c22f2d06=0}, log id: 5a6c565d >2012-09-02 16:32:54,556 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-38) [5cfe1c0b] Host green-vdsa.qa.lab.tlv.redhat.com storage connection was succeeded >2012-09-02 16:32:54,573 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-38) [5cfe1c0b] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 3669e71f >2012-09-02 16:32:57,144 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-38) [5cfe1c0b] FINISH, ConnectStoragePoolVDSCommand, log id: 3669e71f >2012-09-02 16:32:57,148 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-38) [5cfe1c0b] No string for UNASSIGNED type. Use default Log >2012-09-02 16:32:57,151 INFO [org.ovirt.engine.core.bll.MultipleActionsRunner] (pool-4-thread-48) [5cfe1c0b] MultipleActionsRunner of type MigrateVmToServer invoked with no actions >2012-09-02 16:32:57,228 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-38) [74b2dbff] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: b32eacf0-f4e9-11e1-973c-001a4a169705 Type: VDS >2012-09-02 16:32:57,245 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-38) [70e284ba] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: b32eacf0-f4e9-11e1-973c-001a4a169705 Type: VDS >2012-09-02 16:35:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-29) Autorecovering hosts is disabled, skipping >2012-09-02 16:35:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-29) Autorecovering storage domains is disabled, skipping >2012-09-02 16:36:39,064 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8009-7) Checking if user admin@internal is an admin, result true >2012-09-02 16:36:39,064 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8009-7) Running command: LoginUserCommand internal: false. >2012-09-02 16:36:39,092 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8009-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2012-09-02 16:36:39,289 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8009-11) Checking if user admin@internal is an admin, result true >2012-09-02 16:36:39,289 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8009-11) Running command: LoginUserCommand internal: false. >2012-09-02 16:36:39,319 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8009-11) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2012-09-02 16:38:15,727 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8009-10) Checking if user admin@internal is an admin, result true >2012-09-02 16:38:15,727 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8009-10) Running command: LoginUserCommand internal: false. >2012-09-02 16:38:15,781 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8009-10) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2012-09-02 16:38:15,982 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8009-8) Checking if user admin@internal is an admin, result true >2012-09-02 16:38:15,982 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8009-8) Running command: LoginUserCommand internal: false. >2012-09-02 16:38:16,008 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8009-8) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2012-09-02 16:38:50,438 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [70e284ba] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 16:39:18,870 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-92) ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = 3880c572-f4f0-11e1-9a52-001a4a169705 : camel-vdsb.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 16:39:20,877 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-93) [43d82869] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = 3880c572-f4f0-11e1-9a52-001a4a169705 : camel-vdsb.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 16:39:22,884 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-96) [239c5a4] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = 3880c572-f4f0-11e1-9a52-001a4a169705 : camel-vdsb.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 16:39:23,464 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-98) [1fab286a] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: bd560b80-c245-46b3-ad8c-b142a3460cf6 Type: StoragePool >2012-09-02 16:39:23,509 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [1fab286a] IrsBroker::Failed::GetStoragePoolInfoVDS due to: ConnectException: Connection refused >2012-09-02 16:39:23,514 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [1fab286a] Irs placed on server null failed. Proceed Failover >2012-09-02 16:39:23,536 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [1fab286a] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Free, storage pool TestDataCenter >2012-09-02 16:39:23,539 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [1fab286a] SPM Init: could not find reported vds or not up - pool:TestDataCenter vds_spm_id: 2 >2012-09-02 16:39:23,540 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [1fab286a] SPM selection - vds seems as spm camel-vdsb.qa.lab.tlv.redhat.com >2012-09-02 16:39:23,541 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-98) [1fab286a] START, SpmStopVDSCommand(vdsId = 3880c572-f4f0-11e1-9a52-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6), log id: 499113e5 >2012-09-02 16:39:23,541 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-98) [1fab286a] SpmStopVDSCommand:: vds camel-vdsb.qa.lab.tlv.redhat.com is in Connecting status - not performing spm stop, pool id bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 16:39:23,541 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-98) [1fab286a] FINISH, SpmStopVDSCommand, log id: 499113e5 >2012-09-02 16:39:23,541 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [1fab286a] spm stop on spm failed, stopping spm selection! >2012-09-02 16:39:24,893 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-94) [30f99192] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = 3880c572-f4f0-11e1-9a52-001a4a169705 : camel-vdsb.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 16:39:26,900 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-1) [1f04073e] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = 3880c572-f4f0-11e1-9a52-001a4a169705 : camel-vdsb.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 16:39:28,908 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-4) [1b9aaac2] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = 3880c572-f4f0-11e1-9a52-001a4a169705 : camel-vdsb.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 16:39:30,915 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-7) ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = 3880c572-f4f0-11e1-9a52-001a4a169705 : camel-vdsb.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 16:39:32,927 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-6) ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = 3880c572-f4f0-11e1-9a52-001a4a169705 : camel-vdsb.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 16:39:33,570 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Free, storage pool TestDataCenter >2012-09-02 16:39:33,573 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) SPM Init: could not find reported vds or not up - pool:TestDataCenter vds_spm_id: 2 >2012-09-02 16:39:33,574 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) SPM selection - vds seems as spm camel-vdsb.qa.lab.tlv.redhat.com >2012-09-02 16:39:33,575 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-13) START, SpmStopVDSCommand(vdsId = 3880c572-f4f0-11e1-9a52-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6), log id: 7499684b >2012-09-02 16:39:33,575 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-13) SpmStopVDSCommand:: vds camel-vdsb.qa.lab.tlv.redhat.com is in Connecting status - not performing spm stop, pool id bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 16:39:33,575 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-13) FINISH, SpmStopVDSCommand, log id: 7499684b >2012-09-02 16:39:33,575 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) spm stop on spm failed, stopping spm selection! >2012-09-02 16:39:34,934 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-11) [13caede9] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = 3880c572-f4f0-11e1-9a52-001a4a169705 : camel-vdsb.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 16:39:36,941 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-16) [15c45473] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = 3880c572-f4f0-11e1-9a52-001a4a169705 : camel-vdsb.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 16:39:38,949 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-14) [2ff7a817] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = 3880c572-f4f0-11e1-9a52-001a4a169705 : camel-vdsb.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 16:39:40,957 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-17) [7445a7b7] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = 3880c572-f4f0-11e1-9a52-001a4a169705 : camel-vdsb.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 16:39:42,965 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-29) ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = 3880c572-f4f0-11e1-9a52-001a4a169705 : camel-vdsb.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 16:39:43,598 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) [227b1500] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Free, storage pool TestDataCenter >2012-09-02 16:39:43,600 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) [227b1500] SPM Init: could not find reported vds or not up - pool:TestDataCenter vds_spm_id: 2 >2012-09-02 16:39:43,601 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) [227b1500] SPM selection - vds seems as spm camel-vdsb.qa.lab.tlv.redhat.com >2012-09-02 16:39:43,602 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-31) [227b1500] START, SpmStopVDSCommand(vdsId = 3880c572-f4f0-11e1-9a52-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6), log id: 13164e06 >2012-09-02 16:39:43,602 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-31) [227b1500] SpmStopVDSCommand:: vds camel-vdsb.qa.lab.tlv.redhat.com is in Connecting status - not performing spm stop, pool id bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 16:39:43,602 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-31) [227b1500] FINISH, SpmStopVDSCommand, log id: 13164e06 >2012-09-02 16:39:43,602 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) [227b1500] spm stop on spm failed, stopping spm selection! >2012-09-02 16:39:44,972 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-27) [a89e0e7] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = 3880c572-f4f0-11e1-9a52-001a4a169705 : camel-vdsb.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 16:39:46,980 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-33) ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = 3880c572-f4f0-11e1-9a52-001a4a169705 : camel-vdsb.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 16:39:48,988 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-26) [285a1abc] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = 3880c572-f4f0-11e1-9a52-001a4a169705 : camel-vdsb.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 16:39:50,999 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-36) [6c7e8a86] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = 3880c572-f4f0-11e1-9a52-001a4a169705 : camel-vdsb.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 16:39:53,006 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-37) [14f6fa8e] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = 3880c572-f4f0-11e1-9a52-001a4a169705 : camel-vdsb.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 16:39:53,625 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) [2a140678] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Free, storage pool TestDataCenter >2012-09-02 16:39:53,627 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) [2a140678] SPM Init: could not find reported vds or not up - pool:TestDataCenter vds_spm_id: 2 >2012-09-02 16:39:53,628 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) [2a140678] SPM selection - vds seems as spm camel-vdsb.qa.lab.tlv.redhat.com >2012-09-02 16:39:53,629 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-41) [2a140678] START, SpmStopVDSCommand(vdsId = 3880c572-f4f0-11e1-9a52-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6), log id: 3e9b5bb2 >2012-09-02 16:39:53,629 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-41) [2a140678] SpmStopVDSCommand:: vds camel-vdsb.qa.lab.tlv.redhat.com is in Connecting status - not performing spm stop, pool id bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 16:39:53,629 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-41) [2a140678] FINISH, SpmStopVDSCommand, log id: 3e9b5bb2 >2012-09-02 16:39:53,629 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) [2a140678] spm stop on spm failed, stopping spm selection! >2012-09-02 16:39:55,013 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-46) ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = 3880c572-f4f0-11e1-9a52-001a4a169705 : camel-vdsb.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 16:39:57,020 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-49) [767e356a] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = 3880c572-f4f0-11e1-9a52-001a4a169705 : camel-vdsb.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 16:39:59,027 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-48) [1ea6b2c6] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = 3880c572-f4f0-11e1-9a52-001a4a169705 : camel-vdsb.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 16:40:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-60) [dbe32c9] Autorecovering hosts is disabled, skipping >2012-09-02 16:40:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-60) [dbe32c9] Autorecovering storage domains is disabled, skipping >2012-09-02 16:40:01,034 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-44) [30ec8757] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = 3880c572-f4f0-11e1-9a52-001a4a169705 : camel-vdsb.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 16:40:03,041 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-61) ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = 3880c572-f4f0-11e1-9a52-001a4a169705 : camel-vdsb.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 16:40:03,652 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [6dc69ea2] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Free, storage pool TestDataCenter >2012-09-02 16:40:03,654 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [6dc69ea2] SPM Init: could not find reported vds or not up - pool:TestDataCenter vds_spm_id: 2 >2012-09-02 16:40:03,655 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [6dc69ea2] SPM selection - vds seems as spm camel-vdsb.qa.lab.tlv.redhat.com >2012-09-02 16:40:03,656 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-50) [6dc69ea2] START, SpmStopVDSCommand(vdsId = 3880c572-f4f0-11e1-9a52-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6), log id: 7e2969df >2012-09-02 16:40:03,656 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-50) [6dc69ea2] SpmStopVDSCommand:: vds camel-vdsb.qa.lab.tlv.redhat.com is in Connecting status - not performing spm stop, pool id bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 16:40:03,656 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-50) [6dc69ea2] FINISH, SpmStopVDSCommand, log id: 7e2969df >2012-09-02 16:40:03,656 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [6dc69ea2] spm stop on spm failed, stopping spm selection! >2012-09-02 16:40:05,050 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-34) [39d95b35] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = 3880c572-f4f0-11e1-9a52-001a4a169705 : camel-vdsb.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 16:40:07,072 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-55) [1bbf829c] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = 3880c572-f4f0-11e1-9a52-001a4a169705 : camel-vdsb.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 16:40:09,079 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-67) [25620c1f] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = 3880c572-f4f0-11e1-9a52-001a4a169705 : camel-vdsb.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 16:40:11,086 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-22) [3b584081] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = 3880c572-f4f0-11e1-9a52-001a4a169705 : camel-vdsb.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 16:40:13,092 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-57) [6af76567] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = 3880c572-f4f0-11e1-9a52-001a4a169705 : camel-vdsb.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 16:40:13,691 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-75) [2ec70fed] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Free, storage pool TestDataCenter >2012-09-02 16:40:13,693 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-75) [2ec70fed] SPM Init: could not find reported vds or not up - pool:TestDataCenter vds_spm_id: 2 >2012-09-02 16:40:13,694 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-75) [2ec70fed] SPM selection - vds seems as spm camel-vdsb.qa.lab.tlv.redhat.com >2012-09-02 16:40:13,695 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-75) [2ec70fed] START, SpmStopVDSCommand(vdsId = 3880c572-f4f0-11e1-9a52-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6), log id: 3a7f0aae >2012-09-02 16:40:13,695 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-75) [2ec70fed] SpmStopVDSCommand:: vds camel-vdsb.qa.lab.tlv.redhat.com is in Connecting status - not performing spm stop, pool id bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 16:40:13,695 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-75) [2ec70fed] FINISH, SpmStopVDSCommand, log id: 3a7f0aae >2012-09-02 16:40:13,695 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-75) [2ec70fed] spm stop on spm failed, stopping spm selection! >2012-09-02 16:40:15,106 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-59) [4f91000b] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = 3880c572-f4f0-11e1-9a52-001a4a169705 : camel-vdsb.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 16:40:17,114 ERROR [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-84) [17f32511] VDS::handleNetworkException Server failed to respond, vds_id = 3880c572-f4f0-11e1-9a52-001a4a169705, vds_name = camel-vdsb.qa.lab.tlv.redhat.com, error = VDSNetworkException: >2012-09-02 16:40:17,125 INFO [org.ovirt.engine.core.bll.VdsEventListener] (pool-4-thread-48) [17f32511] ResourceManager::vdsNotResponding entered for Host 3880c572-f4f0-11e1-9a52-001a4a169705, 10.35.116.2 >2012-09-02 16:40:17,187 WARN [org.ovirt.engine.core.bll.VdsNotRespondingTreatmentCommand] (pool-4-thread-48) [9bd1f40] CanDoAction of action VdsNotRespondingTreatment failed. Reasons:VDS_FENCING_DISABLED >2012-09-02 16:40:23,717 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [71797221] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Free, storage pool TestDataCenter >2012-09-02 16:40:23,719 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [71797221] SPM Init: could not find reported vds or not up - pool:TestDataCenter vds_spm_id: 2 >2012-09-02 16:40:23,720 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [71797221] SPM selection - vds seems as spm camel-vdsb.qa.lab.tlv.redhat.com >2012-09-02 16:40:23,720 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [71797221] spm vds is non responsive, stopping spm selection. >2012-09-02 16:40:30,377 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-93) [14186f30] Running command: InitVdsOnUpCommand internal: true. >2012-09-02 16:40:30,468 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (QuartzScheduler_Worker-93) [7d422c81] START, ValidateStorageServerConnectionVDSCommand(vdsId = 3880c572-f4f0-11e1-9a52-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, storageType = ISCSI, connectionList = [{ id: 90cee3d3-f602-49f9-8f17-8665c22f2d06, connection: 10.35.64.10 };]), log id: 79a4951a >2012-09-02 16:40:30,491 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (QuartzScheduler_Worker-93) [7d422c81] FINISH, ValidateStorageServerConnectionVDSCommand, return: {90cee3d3-f602-49f9-8f17-8665c22f2d06=0}, log id: 79a4951a >2012-09-02 16:40:30,491 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-93) [7d422c81] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: bd560b80-c245-46b3-ad8c-b142a3460cf6 Type: StoragePool >2012-09-02 16:40:30,493 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (QuartzScheduler_Worker-93) [7d422c81] START, ConnectStorageServerVDSCommand(vdsId = 3880c572-f4f0-11e1-9a52-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, storageType = ISCSI, connectionList = [{ id: 90cee3d3-f602-49f9-8f17-8665c22f2d06, connection: 10.35.64.10 };]), log id: 78cd391a >2012-09-02 16:40:30,574 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (QuartzScheduler_Worker-93) [7d422c81] FINISH, ConnectStorageServerVDSCommand, return: {90cee3d3-f602-49f9-8f17-8665c22f2d06=0}, log id: 78cd391a >2012-09-02 16:40:30,575 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-93) [7d422c81] Host camel-vdsb.qa.lab.tlv.redhat.com storage connection was succeeded >2012-09-02 16:40:30,599 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-93) [7d422c81] START, ConnectStoragePoolVDSCommand(vdsId = 3880c572-f4f0-11e1-9a52-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 2, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 5ffe84f2 >2012-09-02 16:40:31,024 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-93) [7d422c81] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 16:40:31,025 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-93) [7d422c81] Vds: camel-vdsb.qa.lab.tlv.redhat.com >2012-09-02 16:40:31,025 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-93) [7d422c81] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 16:40:31,025 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-93) [7d422c81] FINISH, ConnectStoragePoolVDSCommand, log id: 5ffe84f2 >2012-09-02 16:40:31,025 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-93) [7d422c81] Could not connect host camel-vdsb.qa.lab.tlv.redhat.com to pool TestDataCenter >2012-09-02 16:40:31,027 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-93) [3eca5d1a] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 3880c572-f4f0-11e1-9a52-001a4a169705 Type: VDS >2012-09-02 16:40:31,027 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-93) [3eca5d1a] START, SetVdsStatusVDSCommand(vdsId = 3880c572-f4f0-11e1-9a52-001a4a169705, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 2ad93047 >2012-09-02 16:40:31,029 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-93) [3eca5d1a] SetVdsStatusVDSCommand::VSD camel-vdsb.qa.lab.tlv.redhat.com is spm and moved from up calling ResetIrs. >2012-09-02 16:40:31,029 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-93) [3eca5d1a] START, ResetIrsVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = 10.35.116.2, vdsId = 3880c572-f4f0-11e1-9a52-001a4a169705, ignoreStopFailed = false), log id: 479168d1 >2012-09-02 16:40:31,031 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-93) [3eca5d1a] START, SpmStopVDSCommand(vdsId = 3880c572-f4f0-11e1-9a52-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6), log id: 1d6d815a >2012-09-02 16:40:31,053 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-93) [3eca5d1a] SpmStopVDSCommand::Stopping SPM on vds camel-vdsb.qa.lab.tlv.redhat.com, pool id bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 16:40:31,080 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-93) [3eca5d1a] FINISH, SpmStopVDSCommand, log id: 1d6d815a >2012-09-02 16:40:31,082 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-93) [3eca5d1a] FINISH, ResetIrsVDSCommand, log id: 479168d1 >2012-09-02 16:40:31,082 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-93) [3eca5d1a] FINISH, SetVdsStatusVDSCommand, log id: 2ad93047 >2012-09-02 16:40:31,084 ERROR [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (pool-4-thread-48) [3eca5d1a] Failed to create ExecutionContext for MigrateVmCommand: java.lang.NullPointerException > at org.ovirt.engine.core.bll.MaintananceVdsCommand.createMigrateVmContext(MaintananceVdsCommand.java:114) [engine-bll.jar:] > at org.ovirt.engine.core.bll.MaintananceVdsCommand.MigrateAllVms(MaintananceVdsCommand.java:93) [engine-bll.jar:] > at org.ovirt.engine.core.bll.MaintananceVdsCommand.MigrateAllVms(MaintananceVdsCommand.java:79) [engine-bll.jar:] > at org.ovirt.engine.core.bll.SetNonOperationalVdsCommand$1.run(SetNonOperationalVdsCommand.java:44) [engine-bll.jar:] > at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalWrapperRunnable.run(ThreadPoolUtil.java:64) [engine-utils.jar:] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_05-icedtea] > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) [rt.jar:1.7.0_05-icedtea] > at java.util.concurrent.FutureTask.run(FutureTask.java:166) [rt.jar:1.7.0_05-icedtea] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [rt.jar:1.7.0_05-icedtea] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [rt.jar:1.7.0_05-icedtea] > at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_05-icedtea] > >2012-09-02 16:40:31,196 INFO [org.ovirt.engine.core.bll.MultipleActionsRunner] (pool-4-thread-49) [3eca5d1a] MultipleActionsRunner of type MigrateVmToServer invoked with no actions >2012-09-02 16:40:31,200 INFO [org.ovirt.engine.core.bll.InternalMigrateVmCommand] (pool-4-thread-48) [3cc58346] Running command: InternalMigrateVmCommand internal: true. Entities affected : ID: 5449d6cf-0e31-44b5-b0a9-65036f122662 Type: VM >2012-09-02 16:40:31,210 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-93) [31ddd11f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 3880c572-f4f0-11e1-9a52-001a4a169705 Type: VDS >2012-09-02 16:40:31,215 INFO [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand] (pool-4-thread-48) [3cc58346] START, MigrateVDSCommand(vdsId = 3880c572-f4f0-11e1-9a52-001a4a169705, vmId=5449d6cf-0e31-44b5-b0a9-65036f122662, srcHost=10.35.116.2, dstVdsId=b32eacf0-f4e9-11e1-973c-001a4a169705, dstHost=10.35.102.10:54321, migrationMethod=ONLINE), log id: 55b57e85 >2012-09-02 16:40:31,223 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-93) [6cfa5fec] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 3880c572-f4f0-11e1-9a52-001a4a169705 Type: VDS >2012-09-02 16:40:31,225 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-93) [6cfa5fec] Host 3880c572-f4f0-11e1-9a52-001a4a169705 : camel-vdsb.qa.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2012-09-02 16:40:31,226 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] (pool-4-thread-48) [3cc58346] VdsBroker::migrate::Entered (vm_guid=5449d6cf-0e31-44b5-b0a9-65036f122662, srcHost=10.35.116.2, dstHost=10.35.102.10:54321, method=online >2012-09-02 16:40:31,226 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] (pool-4-thread-48) [3cc58346] START, MigrateBrokerVDSCommand(vdsId = 3880c572-f4f0-11e1-9a52-001a4a169705, vmId=5449d6cf-0e31-44b5-b0a9-65036f122662, srcHost=10.35.116.2, dstVdsId=b32eacf0-f4e9-11e1-973c-001a4a169705, dstHost=10.35.102.10:54321, migrationMethod=ONLINE), log id: 51c214b7 >2012-09-02 16:40:31,248 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] (pool-4-thread-48) [3cc58346] FINISH, MigrateBrokerVDSCommand, log id: 51c214b7 >2012-09-02 16:40:31,260 INFO [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand] (pool-4-thread-48) [3cc58346] FINISH, MigrateVDSCommand, return: MigratingFrom, log id: 55b57e85 >2012-09-02 16:40:33,552 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (QuartzScheduler_Worker-98) [1fab286a] START, DestroyVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, vmId=5449d6cf-0e31-44b5-b0a9-65036f122662, force=false, secondsToWait=0, gracefully=false), log id: 73796609 >2012-09-02 16:40:33,617 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (QuartzScheduler_Worker-98) [1fab286a] FINISH, DestroyVDSCommand, log id: 73796609 >2012-09-02 16:40:33,635 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-98) [1fab286a] vds::refreshVmList vm id 5449d6cf-0e31-44b5-b0a9-65036f122662 status = Down on vds green-vdsa.qa.lab.tlv.redhat.com ignoring it in the refresh till migration is done >2012-09-02 16:40:33,743 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [30f99192] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Free, storage pool TestDataCenter >2012-09-02 16:40:33,769 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [30f99192] SpmStatus on vds 3880c572-f4f0-11e1-9a52-001a4a169705: Unknown_Pool >2012-09-02 16:40:33,779 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [30f99192] starting spm on vds green-vdsa.qa.lab.tlv.redhat.com, storage pool TestDataCenter, prevId 2, LVER 2 >2012-09-02 16:40:33,780 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [30f99192] START, SpmStartVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, prevId=2, prevLVER=2, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2a4090e2 >2012-09-02 16:40:33,802 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [30f99192] spmStart polling started: taskId = 058a7f66-d6d7-4862-a11e-f4a16da9c49f >2012-09-02 16:40:34,824 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-94) [30f99192] Failed in HSMGetTaskStatusVDS method >2012-09-02 16:40:34,824 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-94) [30f99192] Error code GeneralException and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Failed reload: 56a64ce1-90ab-4576-86fa-c269160aaa38 >2012-09-02 16:40:34,824 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [30f99192] spmStart polling ended: taskId = 058a7f66-d6d7-4862-a11e-f4a16da9c49f task status = finished >2012-09-02 16:40:34,824 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [30f99192] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Failed reload: 56a64ce1-90ab-4576-86fa-c269160aaa38 >2012-09-02 16:40:35,076 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-94) [30f99192] Failed in SpmStatusVDS method >2012-09-02 16:40:35,076 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-94) [30f99192] Error code GeneralException and error message VDSGenericException: VDSErrorException: Failed to SpmStatusVDS, error = Failed reload: 56a64ce1-90ab-4576-86fa-c269160aaa38 >2012-09-02 16:40:35,076 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-94) [30f99192] Command SpmStatusVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to SpmStatusVDS, error = Failed reload: 56a64ce1-90ab-4576-86fa-c269160aaa38 >2012-09-02 16:40:35,077 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-94) [30f99192] Command org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.irsbroker.OneUuidReturnForXmlRpc >mUuid 058a7f66-d6d7-4862-a11e-f4a16da9c49f >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 0 >mMessage OK > > >2012-09-02 16:40:35,077 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-94) [30f99192] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 16:40:35,077 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-94) [30f99192] Failed in SpmStartVDS method, for vds: green-vdsa.qa.lab.tlv.redhat.com; host: 10.35.102.10 >2012-09-02 16:40:35,077 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-94) [30f99192] Command SpmStartVDS execution failed. Exception: NullPointerException: >2012-09-02 16:40:35,077 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [30f99192] FINISH, SpmStartVDSCommand, log id: 2a4090e2 >2012-09-02 16:40:35,079 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-94) [59ece3a8] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: bd560b80-c245-46b3-ad8c-b142a3460cf6 Type: StoragePool >2012-09-02 16:40:35,184 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [59ece3a8] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-09-02 16:40:45,463 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-12) [5064a4c0] Failed in SpmStatusVDS method >2012-09-02 16:40:45,463 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-12) [5064a4c0] Error code GeneralException and error message VDSGenericException: VDSErrorException: Failed to SpmStatusVDS, error = Failed reload: 56a64ce1-90ab-4576-86fa-c269160aaa38 >2012-09-02 16:40:45,463 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-12) [5064a4c0] Command SpmStatusVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to SpmStatusVDS, error = Failed reload: 56a64ce1-90ab-4576-86fa-c269160aaa38 >2012-09-02 16:40:45,463 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-12) [5064a4c0] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 16:40:55,713 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-28) [164d9fbc] Failed in SpmStatusVDS method >2012-09-02 16:40:55,713 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-28) [164d9fbc] Error code GeneralException and error message VDSGenericException: VDSErrorException: Failed to SpmStatusVDS, error = Failed reload: 56a64ce1-90ab-4576-86fa-c269160aaa38 >2012-09-02 16:40:55,713 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-28) [164d9fbc] Command SpmStatusVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to SpmStatusVDS, error = Failed reload: 56a64ce1-90ab-4576-86fa-c269160aaa38 >2012-09-02 16:40:55,713 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-28) [164d9fbc] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 16:41:04,054 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-41) [2a140678] VM testVm 5449d6cf-0e31-44b5-b0a9-65036f122662 moved from MigratingFrom --> Up >2012-09-02 16:41:04,054 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-41) [2a140678] adding VM 5449d6cf-0e31-44b5-b0a9-65036f122662 to re-run list >2012-09-02 16:41:04,075 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-41) [2a140678] Rerun vm 5449d6cf-0e31-44b5-b0a9-65036f122662. Called from vds camel-vdsb.qa.lab.tlv.redhat.com >2012-09-02 16:41:05,963 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-53) Failed in SpmStatusVDS method >2012-09-02 16:41:05,963 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-53) Error code GeneralException and error message VDSGenericException: VDSErrorException: Failed to SpmStatusVDS, error = Failed reload: 56a64ce1-90ab-4576-86fa-c269160aaa38 >2012-09-02 16:41:05,963 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-53) Command SpmStatusVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to SpmStatusVDS, error = Failed reload: 56a64ce1-90ab-4576-86fa-c269160aaa38 >2012-09-02 16:41:05,963 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-53) hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 16:41:16,212 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-58) Failed in SpmStatusVDS method >2012-09-02 16:41:16,212 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-58) Error code GeneralException and error message VDSGenericException: VDSErrorException: Failed to SpmStatusVDS, error = Failed reload: 56a64ce1-90ab-4576-86fa-c269160aaa38 >2012-09-02 16:41:16,212 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-58) Command SpmStatusVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to SpmStatusVDS, error = Failed reload: 56a64ce1-90ab-4576-86fa-c269160aaa38 >2012-09-02 16:41:16,213 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-58) hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 16:41:26,433 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-64) [5bdf963c] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Free, storage pool TestDataCenter >2012-09-02 16:41:26,806 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-64) [5bdf963c] SpmStatus on vds 3880c572-f4f0-11e1-9a52-001a4a169705: Unknown_Pool >2012-09-02 16:41:26,886 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-64) [5bdf963c] starting spm on vds green-vdsa.qa.lab.tlv.redhat.com, storage pool TestDataCenter, prevId 2, LVER 2 >2012-09-02 16:41:26,888 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-64) [5bdf963c] START, SpmStartVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, prevId=2, prevLVER=2, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 26f7880a >2012-09-02 16:41:26,917 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-64) [5bdf963c] spmStart polling started: taskId = 5d365215-a705-4e08-a8fb-501fb0a34196 >2012-09-02 16:41:49,036 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (ajp-/127.0.0.1:8009-2) [4272a3f0] Lock Acquired to object EngineLock [exclusiveLocks= key: 3880c572-f4f0-11e1-9a52-001a4a169705 value: VDS >, sharedLocks= ] >2012-09-02 16:41:49,081 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (ajp-/127.0.0.1:8009-2) [4272a3f0] Running command: ActivateVdsCommand internal: false. Entities affected : ID: 3880c572-f4f0-11e1-9a52-001a4a169705 Type: VDS >2012-09-02 16:41:49,083 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp-/127.0.0.1:8009-2) [4272a3f0] START, SetVdsStatusVDSCommand(vdsId = 3880c572-f4f0-11e1-9a52-001a4a169705, status=Unassigned, nonOperationalReason=NONE), log id: 5061b213 >2012-09-02 16:41:49,085 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp-/127.0.0.1:8009-2) [4272a3f0] FINISH, SetVdsStatusVDSCommand, log id: 5061b213 >2012-09-02 16:41:49,126 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (ajp-/127.0.0.1:8009-2) [4272a3f0] START, ActivateVdsVDSCommand(vdsId = 3880c572-f4f0-11e1-9a52-001a4a169705), log id: 25a5127d >2012-09-02 16:41:49,782 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (ajp-/127.0.0.1:8009-2) [17836e05] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 3880c572-f4f0-11e1-9a52-001a4a169705 Type: VDS >2012-09-02 16:41:49,784 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (ajp-/127.0.0.1:8009-2) [46e7cf9] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 3880c572-f4f0-11e1-9a52-001a4a169705 Type: VDS >2012-09-02 16:41:49,784 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (ajp-/127.0.0.1:8009-2) [46e7cf9] FINISH, ActivateVdsVDSCommand, log id: 25a5127d >2012-09-02 16:41:50,206 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (ajp-/127.0.0.1:8009-2) [46e7cf9] Lock freed to object EngineLock [exclusiveLocks= key: 3880c572-f4f0-11e1-9a52-001a4a169705 value: VDS >, sharedLocks= ] >2012-09-02 16:41:50,614 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-5) [3621461a] Running command: InitVdsOnUpCommand internal: true. >2012-09-02 16:41:51,093 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (QuartzScheduler_Worker-5) [137d6357] START, ValidateStorageServerConnectionVDSCommand(vdsId = 3880c572-f4f0-11e1-9a52-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, storageType = ISCSI, connectionList = [{ id: 90cee3d3-f602-49f9-8f17-8665c22f2d06, connection: 10.35.64.10 };]), log id: 585f1258 >2012-09-02 16:41:51,115 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (QuartzScheduler_Worker-5) [137d6357] FINISH, ValidateStorageServerConnectionVDSCommand, return: {90cee3d3-f602-49f9-8f17-8665c22f2d06=0}, log id: 585f1258 >2012-09-02 16:41:51,115 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-5) [137d6357] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: bd560b80-c245-46b3-ad8c-b142a3460cf6 Type: StoragePool >2012-09-02 16:41:51,116 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (QuartzScheduler_Worker-5) [137d6357] START, ConnectStorageServerVDSCommand(vdsId = 3880c572-f4f0-11e1-9a52-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, storageType = ISCSI, connectionList = [{ id: 90cee3d3-f602-49f9-8f17-8665c22f2d06, connection: 10.35.64.10 };]), log id: 3c449571 >2012-09-02 16:41:51,224 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (QuartzScheduler_Worker-5) [137d6357] FINISH, ConnectStorageServerVDSCommand, return: {90cee3d3-f602-49f9-8f17-8665c22f2d06=0}, log id: 3c449571 >2012-09-02 16:41:51,224 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-5) [137d6357] Host camel-vdsb.qa.lab.tlv.redhat.com storage connection was succeeded >2012-09-02 16:41:51,239 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-5) [137d6357] START, ConnectStoragePoolVDSCommand(vdsId = 3880c572-f4f0-11e1-9a52-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 2, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 518a3f6e >2012-09-02 16:41:53,636 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-5) [137d6357] FINISH, ConnectStoragePoolVDSCommand, log id: 518a3f6e >2012-09-02 16:41:53,641 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-5) [137d6357] No string for UNASSIGNED type. Use default Log >2012-09-02 16:41:53,643 INFO [org.ovirt.engine.core.bll.MultipleActionsRunner] (pool-4-thread-49) [137d6357] MultipleActionsRunner of type MigrateVmToServer invoked with no actions >2012-09-02 16:41:53,831 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-5) [1b92f647] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 3880c572-f4f0-11e1-9a52-001a4a169705 Type: VDS >2012-09-02 16:41:53,855 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-5) [40eb8ded] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 3880c572-f4f0-11e1-9a52-001a4a169705 Type: VDS >2012-09-02 16:41:55,573 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-64) [5bdf963c] spmStart polling ended: taskId = 5d365215-a705-4e08-a8fb-501fb0a34196 task status = finished >2012-09-02 16:41:55,598 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-64) [5bdf963c] spmStart polling ended. spm status: SPM >2012-09-02 16:41:55,599 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-64) [5bdf963c] START, HSMClearTaskVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, taskId=5d365215-a705-4e08-a8fb-501fb0a34196), log id: 59aea2ba >2012-09-02 16:41:55,617 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-64) [5bdf963c] FINISH, HSMClearTaskVDSCommand, log id: 59aea2ba >2012-09-02 16:41:55,617 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-64) [5bdf963c] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@583d8f55, log id: 26f7880a >2012-09-02 16:41:55,628 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-64) [5bdf963c] Initialize Irs proxy from vds: 10.35.102.10 >2012-09-02 16:41:55,656 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-10) [29507acf] Attempting to get and stop tasks on storage pool TestDataCenter >2012-09-02 16:41:55,656 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-10) [29507acf] START, SPMGetAllTasksInfoVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 52df0a50 >2012-09-02 16:41:55,705 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-10) [29507acf] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 16:41:55,706 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-10) [29507acf] START, HSMGetAllTasksInfoVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705), log id: 4cff3d30 >2012-09-02 16:41:55,725 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-10) [29507acf] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4cff3d30 >2012-09-02 16:41:55,725 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-10) [29507acf] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 52df0a50 >2012-09-02 16:41:55,725 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-10) [29507acf] Discovered no tasks on Storage Pool TestDataCenter >2012-09-02 16:41:55,725 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-10) [29507acf] START, SPMGetAllTasksInfoVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 90a41ce >2012-09-02 16:41:55,725 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-10) [29507acf] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 16:41:55,728 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-10) [29507acf] START, HSMGetAllTasksInfoVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705), log id: 2d8fe2fa >2012-09-02 16:41:55,748 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-10) [29507acf] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 2d8fe2fa >2012-09-02 16:41:55,748 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-10) [29507acf] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 90a41ce >2012-09-02 16:41:55,748 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-10) [29507acf] Discovered no tasks on Storage Pool TestDataCenter >2012-09-02 16:41:56,003 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [52fbf215] Domain 56a64ce1-90ab-4576-86fa-c269160aaa38 recovered from problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 16:42:02,105 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [2ff7a817] Domain 56a64ce1-90ab-4576-86fa-c269160aaa38 recovered from problem. vds: camel-vdsb.qa.lab.tlv.redhat.com >2012-09-02 16:42:02,105 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [2ff7a817] Domain 56a64ce1-90ab-4576-86fa-c269160aaa38 has recovered from problem. No active host in the DC is reporting it as problematic, so clearing the domain recovery timer. >2012-09-02 16:45:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-70) [4679b9c1] Autorecovering hosts is disabled, skipping >2012-09-02 16:45:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-70) [4679b9c1] Autorecovering storage domains is disabled, skipping >2012-09-02 16:45:08,086 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8009-7) Checking if user admin@internal is an admin, result true >2012-09-02 16:45:08,086 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8009-7) Running command: LoginUserCommand internal: false. >2012-09-02 16:45:08,173 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8009-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2012-09-02 16:45:08,373 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8009-1) Checking if user admin@internal is an admin, result true >2012-09-02 16:45:08,373 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8009-1) Running command: LoginUserCommand internal: false. >2012-09-02 16:45:08,417 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8009-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2012-09-02 16:45:40,759 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [4722d115] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: camel-vdsb.qa.lab.tlv.redhat.com >2012-09-02 16:46:13,176 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-76) [a637d5a] vds::refreshVdsStats Failed getVdsStats, vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, error = VDSNetworkException: VDSNetworkException: >2012-09-02 16:46:13,194 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-76) [a637d5a] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 16:46:15,208 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-82) [f21968f] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 16:46:17,216 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-90) [5150576a] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 16:46:19,225 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-89) [5fb3a86f] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 16:46:21,232 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-85) [5e430da1] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 16:46:21,411 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-91) [14e5a60b] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: bd560b80-c245-46b3-ad8c-b142a3460cf6 Type: StoragePool >2012-09-02 16:46:21,775 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-91) [14e5a60b] IrsBroker::Failed::GetStoragePoolInfoVDS due to: ConnectException: Connection refused >2012-09-02 16:46:21,783 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-91) [14e5a60b] Irs placed on server null failed. Proceed Failover >2012-09-02 16:46:23,243 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-83) [f848894] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 16:46:25,250 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-96) [239c5a4] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 16:46:27,261 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-94) [59ece3a8] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 16:46:29,268 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-97) [55a6af91] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 16:46:31,525 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-1) [1f04073e] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 16:46:33,532 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-4) [1b9aaac2] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 16:46:35,538 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-6) ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 16:46:37,545 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-10) [29507acf] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 16:46:39,553 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-11) [13caede9] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 16:46:41,560 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-18) [64c55214] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 16:46:43,571 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-20) ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 16:46:45,579 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-21) [2adea929] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 16:46:47,588 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-31) [227b1500] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 16:46:49,634 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-27) [a89e0e7] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 16:46:51,642 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-28) [164d9fbc] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 16:46:53,649 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-30) [4722d115] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 16:46:55,655 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-43) [35748f38] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 16:46:57,665 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-37) [14f6fa8e] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 16:46:59,672 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-39) [62249710] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 16:47:01,679 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-46) ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 16:47:03,687 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-45) [4a22f461] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 16:47:05,694 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-56) ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 16:47:07,700 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-44) [30ec8757] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 16:47:09,708 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-61) ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 16:47:11,715 ERROR [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-54) VDS::handleNetworkException Server failed to respond, vds_id = b32eacf0-f4e9-11e1-973c-001a4a169705, vds_name = green-vdsa.qa.lab.tlv.redhat.com, error = VDSNetworkException: >2012-09-02 16:47:11,739 INFO [org.ovirt.engine.core.bll.VdsEventListener] (pool-4-thread-48) ResourceManager::vdsNotResponding entered for Host b32eacf0-f4e9-11e1-973c-001a4a169705, 10.35.102.10 >2012-09-02 16:47:11,785 WARN [org.ovirt.engine.core.bll.VdsNotRespondingTreatmentCommand] (pool-4-thread-48) [67794886] CanDoAction of action VdsNotRespondingTreatment failed. Reasons:VDS_FENCING_DISABLED >2012-09-02 16:47:19,156 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-91) [14e5a60b] Failed in SpmStatusVDS method >2012-09-02 16:47:19,156 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-91) [14e5a60b] Error code GeneralException and error message VDSGenericException: VDSErrorException: Failed to SpmStatusVDS, error = Failed reload: 56a64ce1-90ab-4576-86fa-c269160aaa38 >2012-09-02 16:47:19,156 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-91) [14e5a60b] Command SpmStatusVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to SpmStatusVDS, error = Failed reload: 56a64ce1-90ab-4576-86fa-c269160aaa38 >2012-09-02 16:47:19,156 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-91) [14e5a60b] hostFromVds::selectedVds - camel-vdsb.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 16:47:23,737 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-75) [18c9f1c2] Running command: InitVdsOnUpCommand internal: true. >2012-09-02 16:47:23,840 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (QuartzScheduler_Worker-75) [436bfe0d] START, ValidateStorageServerConnectionVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, storageType = ISCSI, connectionList = [{ id: 90cee3d3-f602-49f9-8f17-8665c22f2d06, connection: 10.35.64.10 };]), log id: 72edbef5 >2012-09-02 16:47:23,872 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (QuartzScheduler_Worker-75) [436bfe0d] FINISH, ValidateStorageServerConnectionVDSCommand, return: {90cee3d3-f602-49f9-8f17-8665c22f2d06=0}, log id: 72edbef5 >2012-09-02 16:47:23,873 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-75) [436bfe0d] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: bd560b80-c245-46b3-ad8c-b142a3460cf6 Type: StoragePool >2012-09-02 16:47:23,874 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (QuartzScheduler_Worker-75) [436bfe0d] START, ConnectStorageServerVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, storageType = ISCSI, connectionList = [{ id: 90cee3d3-f602-49f9-8f17-8665c22f2d06, connection: 10.35.64.10 };]), log id: 56b62736 >2012-09-02 16:47:23,990 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (QuartzScheduler_Worker-75) [436bfe0d] FINISH, ConnectStorageServerVDSCommand, return: {90cee3d3-f602-49f9-8f17-8665c22f2d06=0}, log id: 56b62736 >2012-09-02 16:47:23,990 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-75) [436bfe0d] Host green-vdsa.qa.lab.tlv.redhat.com storage connection was succeeded >2012-09-02 16:47:24,068 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-75) [436bfe0d] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 651dd7ad >2012-09-02 16:47:24,941 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-75) [436bfe0d] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 16:47:24,941 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-75) [436bfe0d] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 16:47:24,941 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-75) [436bfe0d] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 16:47:24,941 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-75) [436bfe0d] FINISH, ConnectStoragePoolVDSCommand, log id: 651dd7ad >2012-09-02 16:47:24,941 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-75) [436bfe0d] Could not connect host green-vdsa.qa.lab.tlv.redhat.com to pool TestDataCenter >2012-09-02 16:47:24,943 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-75) [2b3b663b] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: b32eacf0-f4e9-11e1-973c-001a4a169705 Type: VDS >2012-09-02 16:47:24,943 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-75) [2b3b663b] START, SetVdsStatusVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 5002a523 >2012-09-02 16:47:24,946 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-75) [2b3b663b] SetVdsStatusVDSCommand::VSD green-vdsa.qa.lab.tlv.redhat.com is spm and moved from up calling ResetIrs. >2012-09-02 16:47:24,946 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-75) [2b3b663b] START, ResetIrsVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = 10.35.102.10, vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, ignoreStopFailed = false), log id: 2f422369 >2012-09-02 16:47:24,946 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-75) [2b3b663b] START, SpmStopVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6), log id: 436c8b49 >2012-09-02 16:47:24,966 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-75) [2b3b663b] SpmStopVDSCommand::Stopping SPM on vds green-vdsa.qa.lab.tlv.redhat.com, pool id bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 16:47:24,989 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-75) [2b3b663b] FINISH, SpmStopVDSCommand, log id: 436c8b49 >2012-09-02 16:47:24,994 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-75) [2b3b663b] FINISH, ResetIrsVDSCommand, log id: 2f422369 >2012-09-02 16:47:24,994 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-75) [2b3b663b] FINISH, SetVdsStatusVDSCommand, log id: 5002a523 >2012-09-02 16:47:25,301 INFO [org.ovirt.engine.core.bll.MultipleActionsRunner] (pool-4-thread-48) [2b3b663b] MultipleActionsRunner of type MigrateVmToServer invoked with no actions >2012-09-02 16:47:25,329 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-75) [b876bb3] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: b32eacf0-f4e9-11e1-973c-001a4a169705 Type: VDS >2012-09-02 16:47:25,344 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-75) [1b3d0324] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: b32eacf0-f4e9-11e1-973c-001a4a169705 Type: VDS >2012-09-02 16:47:25,345 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-75) [1b3d0324] Host b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2012-09-02 16:47:29,525 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-72) [2ab07b12] Command SpmStatusVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 16:47:29,525 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [2ab07b12] hostFromVds::selectedVds - camel-vdsb.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 16:47:29,525 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [2ab07b12] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 16:47:29,525 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [2ab07b12] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 16:47:29,597 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-72) [44d07d30] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 16:47:29,630 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-72) [44d07d30] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 33d0b64a >2012-09-02 16:47:29,630 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [44d07d30] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 16:47:29,630 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-72) [44d07d30] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 33d0b64a >2012-09-02 16:47:35,551 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [5e430da1] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: camel-vdsb.qa.lab.tlv.redhat.com >2012-09-02 16:47:39,880 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-95) [4fcef51] Command SpmStatusVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 16:47:39,880 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-95) [4fcef51] hostFromVds::selectedVds - camel-vdsb.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 16:47:39,880 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-95) [4fcef51] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 16:47:39,880 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-95) [4fcef51] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 16:47:40,262 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-95) [4952ea3d] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 16:47:40,320 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-95) [4952ea3d] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 60b6d98 >2012-09-02 16:47:40,320 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-95) [4952ea3d] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 16:47:40,320 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-95) [4952ea3d] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 60b6d98 >2012-09-02 16:47:47,809 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: camel-vdsb.qa.lab.tlv.redhat.com >2012-09-02 16:47:50,549 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-13) Command SpmStatusVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 16:47:50,549 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) hostFromVds::selectedVds - camel-vdsb.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 16:47:50,549 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 16:47:50,549 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 16:47:50,645 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-13) [3cc618be] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 16:47:50,689 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-13) [3cc618be] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7e4a4585 >2012-09-02 16:47:50,690 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) [3cc618be] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 16:47:50,690 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-13) [3cc618be] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7e4a4585 >2012-09-02 16:47:58,758 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [7445a7b7] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: camel-vdsb.qa.lab.tlv.redhat.com >2012-09-02 16:48:00,948 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-21) [2adea929] Command SpmStatusVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 16:48:00,948 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) [2adea929] hostFromVds::selectedVds - camel-vdsb.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 16:48:00,949 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) [2adea929] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 16:48:00,949 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) [2adea929] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 16:48:01,076 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-21) [4b5ea5e5] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 16:48:01,114 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-21) [4b5ea5e5] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 58a990a8 >2012-09-02 16:48:01,114 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) [4b5ea5e5] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 16:48:01,114 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-21) [4b5ea5e5] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 58a990a8 >2012-09-02 16:48:09,030 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-26) [285a1abc] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: camel-vdsb.qa.lab.tlv.redhat.com >2012-09-02 16:48:11,449 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-36) [6c7e8a86] Command SpmStatusVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 16:48:11,449 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-36) [6c7e8a86] hostFromVds::selectedVds - camel-vdsb.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 16:48:11,449 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-36) [6c7e8a86] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 16:48:11,449 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-36) [6c7e8a86] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 16:48:11,551 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-36) [765c3ffe] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 16:48:11,586 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-36) [765c3ffe] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 41b77388 >2012-09-02 16:48:11,586 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-36) [765c3ffe] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 16:48:11,586 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-36) [765c3ffe] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 41b77388 >2012-09-02 16:48:19,321 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [70e284ba] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: camel-vdsb.qa.lab.tlv.redhat.com >2012-09-02 16:48:22,172 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) hostFromVds::selectedVds - camel-vdsb.qa.lab.tlv.redhat.com, spmStatus Free, storage pool TestDataCenter >2012-09-02 16:48:22,218 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) SpmStatus on vds b32eacf0-f4e9-11e1-973c-001a4a169705: Unknown_Pool >2012-09-02 16:48:22,237 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) starting spm on vds camel-vdsb.qa.lab.tlv.redhat.com, storage pool TestDataCenter, prevId 1, LVER 3 >2012-09-02 16:48:22,238 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) START, SpmStartVDSCommand(vdsId = 3880c572-f4f0-11e1-9a52-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, prevId=1, prevLVER=3, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 521669f2 >2012-09-02 16:48:22,264 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) spmStart polling started: taskId = 54e2a7a6-99c9-459d-9384-a787e58e61f7 >2012-09-02 16:48:41,705 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (ajp-/127.0.0.1:8009-1) [50104485] Lock Acquired to object EngineLock [exclusiveLocks= key: b32eacf0-f4e9-11e1-973c-001a4a169705 value: VDS >, sharedLocks= ] >2012-09-02 16:48:41,733 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (ajp-/127.0.0.1:8009-1) [50104485] Running command: ActivateVdsCommand internal: false. Entities affected : ID: b32eacf0-f4e9-11e1-973c-001a4a169705 Type: VDS >2012-09-02 16:48:41,736 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp-/127.0.0.1:8009-1) [50104485] START, SetVdsStatusVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, status=Unassigned, nonOperationalReason=NONE), log id: 1bcc09d6 >2012-09-02 16:48:41,742 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp-/127.0.0.1:8009-1) [50104485] FINISH, SetVdsStatusVDSCommand, log id: 1bcc09d6 >2012-09-02 16:48:41,755 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (ajp-/127.0.0.1:8009-1) [50104485] START, ActivateVdsVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705), log id: 521d9b73 >2012-09-02 16:48:42,057 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (ajp-/127.0.0.1:8009-1) [41e25576] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: b32eacf0-f4e9-11e1-973c-001a4a169705 Type: VDS >2012-09-02 16:48:42,059 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (ajp-/127.0.0.1:8009-1) [31966426] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: b32eacf0-f4e9-11e1-973c-001a4a169705 Type: VDS >2012-09-02 16:48:42,060 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (ajp-/127.0.0.1:8009-1) [31966426] FINISH, ActivateVdsVDSCommand, log id: 521d9b73 >2012-09-02 16:48:42,100 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (ajp-/127.0.0.1:8009-1) [31966426] Lock freed to object EngineLock [exclusiveLocks= key: b32eacf0-f4e9-11e1-973c-001a4a169705 value: VDS >, sharedLocks= ] >2012-09-02 16:48:43,683 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-81) [563bdf09] Running command: InitVdsOnUpCommand internal: true. >2012-09-02 16:48:43,754 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (QuartzScheduler_Worker-81) [362b8462] START, ValidateStorageServerConnectionVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, storageType = ISCSI, connectionList = [{ id: 90cee3d3-f602-49f9-8f17-8665c22f2d06, connection: 10.35.64.10 };]), log id: 54241596 >2012-09-02 16:48:43,774 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (QuartzScheduler_Worker-81) [362b8462] FINISH, ValidateStorageServerConnectionVDSCommand, return: {90cee3d3-f602-49f9-8f17-8665c22f2d06=0}, log id: 54241596 >2012-09-02 16:48:43,774 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-81) [362b8462] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: bd560b80-c245-46b3-ad8c-b142a3460cf6 Type: StoragePool >2012-09-02 16:48:43,775 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (QuartzScheduler_Worker-81) [362b8462] START, ConnectStorageServerVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, storageType = ISCSI, connectionList = [{ id: 90cee3d3-f602-49f9-8f17-8665c22f2d06, connection: 10.35.64.10 };]), log id: 764ec2e1 >2012-09-02 16:48:43,903 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (QuartzScheduler_Worker-81) [362b8462] FINISH, ConnectStorageServerVDSCommand, return: {90cee3d3-f602-49f9-8f17-8665c22f2d06=0}, log id: 764ec2e1 >2012-09-02 16:48:43,903 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-81) [362b8462] Host green-vdsa.qa.lab.tlv.redhat.com storage connection was succeeded >2012-09-02 16:48:43,922 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-81) [362b8462] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 3e601bef >2012-09-02 16:48:46,554 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-81) [362b8462] FINISH, ConnectStoragePoolVDSCommand, log id: 3e601bef >2012-09-02 16:48:46,561 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-81) [362b8462] No string for UNASSIGNED type. Use default Log >2012-09-02 16:48:46,562 INFO [org.ovirt.engine.core.bll.MultipleActionsRunner] (pool-4-thread-49) [362b8462] MultipleActionsRunner of type MigrateVmToServer invoked with no actions >2012-09-02 16:48:46,580 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-81) [28ac90b3] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: b32eacf0-f4e9-11e1-973c-001a4a169705 Type: VDS >2012-09-02 16:48:46,602 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-81) [28d52053] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: b32eacf0-f4e9-11e1-973c-001a4a169705 Type: VDS >2012-09-02 16:48:47,930 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) spmStart polling ended: taskId = 54e2a7a6-99c9-459d-9384-a787e58e61f7 task status = finished >2012-09-02 16:48:47,951 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) spmStart polling ended. spm status: SPM >2012-09-02 16:48:47,952 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-61) START, HSMClearTaskVDSCommand(vdsId = 3880c572-f4f0-11e1-9a52-001a4a169705, taskId=54e2a7a6-99c9-459d-9384-a787e58e61f7), log id: 3651f772 >2012-09-02 16:48:47,972 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-61) FINISH, HSMClearTaskVDSCommand, log id: 3651f772 >2012-09-02 16:48:47,972 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3f4c793a, log id: 521669f2 >2012-09-02 16:48:47,980 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) Initialize Irs proxy from vds: 10.35.116.2 >2012-09-02 16:48:48,017 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-92) Attempting to get and stop tasks on storage pool TestDataCenter >2012-09-02 16:48:48,017 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-92) START, SPMGetAllTasksInfoVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 74708595 >2012-09-02 16:48:48,510 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-92) -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 16:48:48,512 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-92) START, HSMGetAllTasksInfoVDSCommand(vdsId = 3880c572-f4f0-11e1-9a52-001a4a169705), log id: 4cf52c9c >2012-09-02 16:48:48,532 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-92) FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4cf52c9c >2012-09-02 16:48:48,532 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-92) FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 74708595 >2012-09-02 16:48:48,532 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-92) Discovered no tasks on Storage Pool TestDataCenter >2012-09-02 16:48:48,533 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-92) START, SPMGetAllTasksInfoVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 1371c9c6 >2012-09-02 16:48:48,533 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-92) -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 16:48:48,534 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-92) START, HSMGetAllTasksInfoVDSCommand(vdsId = 3880c572-f4f0-11e1-9a52-001a4a169705), log id: 140972aa >2012-09-02 16:48:48,554 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-92) FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 140972aa >2012-09-02 16:48:48,554 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-92) FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 1371c9c6 >2012-09-02 16:48:48,554 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-92) Discovered no tasks on Storage Pool TestDataCenter >2012-09-02 16:50:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-93) [6cfa5fec] Autorecovering hosts is disabled, skipping >2012-09-02 16:50:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-93) [6cfa5fec] Autorecovering storage domains is disabled, skipping >2012-09-02 16:50:40,760 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [70e284ba] starting ProcessDomainRecovery for domain 56a64ce1-90ab-4576-86fa-c269160aaa38 >2012-09-02 16:50:40,763 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [70e284ba] vds camel-vdsb.qa.lab.tlv.redhat.com reported domain 56a64ce1-90ab-4576-86fa-c269160aaa38:ISCSIDataDomain as in problem, moving the vds to status NonOperational >2012-09-02 16:50:40,782 WARN [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (QuartzScheduler_Worker-38) [1a3821] The message key SetNonOperationalVds is missing from bundles/ExecutionMessages >2012-09-02 16:50:40,818 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-38) [1a3821] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 3880c572-f4f0-11e1-9a52-001a4a169705 Type: VDS >2012-09-02 16:50:40,818 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-38) [1a3821] START, SetVdsStatusVDSCommand(vdsId = 3880c572-f4f0-11e1-9a52-001a4a169705, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 1762774f >2012-09-02 16:50:40,820 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-38) [1a3821] SetVdsStatusVDSCommand::VSD camel-vdsb.qa.lab.tlv.redhat.com is spm and moved from up calling ResetIrs. >2012-09-02 16:50:40,820 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-38) [1a3821] START, ResetIrsVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = 10.35.116.2, vdsId = 3880c572-f4f0-11e1-9a52-001a4a169705, ignoreStopFailed = false), log id: 575e58ef >2012-09-02 16:50:40,823 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-38) [1a3821] START, SpmStopVDSCommand(vdsId = 3880c572-f4f0-11e1-9a52-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6), log id: 70b31f39 >2012-09-02 16:50:40,846 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-38) [1a3821] SpmStopVDSCommand::Stopping SPM on vds camel-vdsb.qa.lab.tlv.redhat.com, pool id bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 16:50:41,650 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-38) [1a3821] FINISH, SpmStopVDSCommand, log id: 70b31f39 >2012-09-02 16:50:41,652 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-38) [1a3821] FINISH, ResetIrsVDSCommand, log id: 575e58ef >2012-09-02 16:50:41,652 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-38) [1a3821] FINISH, SetVdsStatusVDSCommand, log id: 1762774f >2012-09-02 16:50:41,764 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [1a3821] Clearing cache of pool: bd560b80-c245-46b3-ad8c-b142a3460cf6 for problematic entities of VDS: camel-vdsb.qa.lab.tlv.redhat.com. >2012-09-02 16:50:41,764 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [1a3821] Domain 56a64ce1-90ab-4576-86fa-c269160aaa38 recovered from problem. vds: camel-vdsb.qa.lab.tlv.redhat.com >2012-09-02 16:50:41,764 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [1a3821] Domain 56a64ce1-90ab-4576-86fa-c269160aaa38 has recovered from problem. No active host in the DC is reporting it as problematic, so clearing the domain recovery timer. >2012-09-02 16:50:41,820 INFO [org.ovirt.engine.core.bll.InternalMigrateVmCommand] (pool-4-thread-50) [4d24a8e6] Running command: InternalMigrateVmCommand internal: true. Entities affected : ID: 5449d6cf-0e31-44b5-b0a9-65036f122662 Type: VM >2012-09-02 16:50:41,827 INFO [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand] (pool-4-thread-50) [4d24a8e6] START, MigrateVDSCommand(vdsId = 3880c572-f4f0-11e1-9a52-001a4a169705, vmId=5449d6cf-0e31-44b5-b0a9-65036f122662, srcHost=10.35.116.2, dstVdsId=b32eacf0-f4e9-11e1-973c-001a4a169705, dstHost=10.35.102.10:54321, migrationMethod=ONLINE), log id: 87e4be0 >2012-09-02 16:50:41,828 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] (pool-4-thread-50) [4d24a8e6] VdsBroker::migrate::Entered (vm_guid=5449d6cf-0e31-44b5-b0a9-65036f122662, srcHost=10.35.116.2, dstHost=10.35.102.10:54321, method=online >2012-09-02 16:50:41,828 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] (pool-4-thread-50) [4d24a8e6] START, MigrateBrokerVDSCommand(vdsId = 3880c572-f4f0-11e1-9a52-001a4a169705, vmId=5449d6cf-0e31-44b5-b0a9-65036f122662, srcHost=10.35.116.2, dstVdsId=b32eacf0-f4e9-11e1-973c-001a4a169705, dstHost=10.35.102.10:54321, migrationMethod=ONLINE), log id: 78641c08 >2012-09-02 16:50:41,849 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] (pool-4-thread-50) [4d24a8e6] FINISH, MigrateBrokerVDSCommand, log id: 78641c08 >2012-09-02 16:50:41,859 INFO [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand] (pool-4-thread-50) [4d24a8e6] FINISH, MigrateVDSCommand, return: MigratingFrom, log id: 87e4be0 >2012-09-02 16:50:44,273 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-50) [6dc69ea2] vds::refreshVmList vm id 5449d6cf-0e31-44b5-b0a9-65036f122662 is migrating to vds green-vdsa.qa.lab.tlv.redhat.com ignoring it in the refresh till migration is done >2012-09-02 16:50:44,601 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) [46ff88ea] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: camel-vdsb.qa.lab.tlv.redhat.com >2012-09-02 16:50:46,358 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (QuartzScheduler_Worker-55) [1bbf829c] START, FullListVdsCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, vds=null, vmIds=[5449d6cf-0e31-44b5-b0a9-65036f122662]), log id: 74a866a0 >2012-09-02 16:50:46,376 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (QuartzScheduler_Worker-55) [1bbf829c] FINISH, FullListVdsCommand, return: [Lorg.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcStruct;@5be9ce35, log id: 74a866a0 >2012-09-02 16:50:46,670 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (QuartzScheduler_Worker-65) [ffaf13b] START, DestroyVDSCommand(vdsId = 3880c572-f4f0-11e1-9a52-001a4a169705, vmId=5449d6cf-0e31-44b5-b0a9-65036f122662, force=false, secondsToWait=0, gracefully=false), log id: 228dced0 >2012-09-02 16:50:46,748 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (QuartzScheduler_Worker-65) [ffaf13b] FINISH, DestroyVDSCommand, log id: 228dced0 >2012-09-02 16:50:46,749 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-65) [ffaf13b] Setting VM testVm 5449d6cf-0e31-44b5-b0a9-65036f122662 to status unknown >2012-09-02 16:50:46,749 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-65) [ffaf13b] vm testVm running in db and not running in vds - add to rerun treatment. vds camel-vdsb.qa.lab.tlv.redhat.com >2012-09-02 16:50:49,552 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-22) [4d76203] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: bd560b80-c245-46b3-ad8c-b142a3460cf6 Type: StoragePool >2012-09-02 16:50:49,852 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-22) [4d76203] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Free, storage pool TestDataCenter >2012-09-02 16:50:49,867 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-22) [4d76203] starting spm on vds green-vdsa.qa.lab.tlv.redhat.com, storage pool TestDataCenter, prevId -1, LVER 3 >2012-09-02 16:50:49,868 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-22) [4d76203] START, SpmStartVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, prevId=-1, prevLVER=3, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4efd3ed9 >2012-09-02 16:50:49,890 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-22) [4d76203] spmStart polling started: taskId = 52408ac8-ee0c-49c2-866d-ff4569f5cbae >2012-09-02 16:50:51,958 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-22) [4d76203] spmStart polling ended: taskId = 52408ac8-ee0c-49c2-866d-ff4569f5cbae task status = finished >2012-09-02 16:50:51,979 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-22) [4d76203] spmStart polling ended. spm status: SPM >2012-09-02 16:50:51,980 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-22) [4d76203] START, HSMClearTaskVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, taskId=52408ac8-ee0c-49c2-866d-ff4569f5cbae), log id: 2053f5b9 >2012-09-02 16:50:51,998 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-22) [4d76203] FINISH, HSMClearTaskVDSCommand, log id: 2053f5b9 >2012-09-02 16:50:51,998 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-22) [4d76203] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@68295543, log id: 4efd3ed9 >2012-09-02 16:50:52,005 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-22) [4d76203] Initialize Irs proxy from vds: 10.35.102.10 >2012-09-02 16:50:52,039 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-59) [4f91000b] Attempting to get and stop tasks on storage pool TestDataCenter >2012-09-02 16:50:52,039 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-59) [4f91000b] START, SPMGetAllTasksInfoVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 5825f2c5 >2012-09-02 16:50:52,083 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-59) [4f91000b] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 16:50:52,084 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-59) [4f91000b] START, HSMGetAllTasksInfoVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705), log id: 527cc03 >2012-09-02 16:50:52,103 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-59) [4f91000b] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 527cc03 >2012-09-02 16:50:52,103 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-59) [4f91000b] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5825f2c5 >2012-09-02 16:50:52,103 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-59) [4f91000b] Discovered no tasks on Storage Pool TestDataCenter >2012-09-02 16:50:52,103 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-59) [4f91000b] START, SPMGetAllTasksInfoVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 77c2db99 >2012-09-02 16:50:52,103 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-59) [4f91000b] -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 16:50:52,104 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-59) [4f91000b] START, HSMGetAllTasksInfoVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705), log id: 6c5de387 >2012-09-02 16:50:52,122 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-59) [4f91000b] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 6c5de387 >2012-09-02 16:50:52,122 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-59) [4f91000b] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 77c2db99 >2012-09-02 16:50:52,122 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-59) [4f91000b] Discovered no tasks on Storage Pool TestDataCenter >2012-09-02 16:52:35,551 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-15) [5147a4f8] starting ProcessDomainRecovery for domain 56a64ce1-90ab-4576-86fa-c269160aaa38 >2012-09-02 16:52:35,555 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-15) [5147a4f8] vds camel-vdsb.qa.lab.tlv.redhat.com reported domain 56a64ce1-90ab-4576-86fa-c269160aaa38:ISCSIDataDomain as in problem, vds is in status NonOperational, no need to move to nonoperational >2012-09-02 16:55:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-24) [392574c] Autorecovering hosts is disabled, skipping >2012-09-02 16:55:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-24) [392574c] Autorecovering storage domains is disabled, skipping >2012-09-02 17:00:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-68) Autorecovering hosts is disabled, skipping >2012-09-02 17:00:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-68) Autorecovering storage domains is disabled, skipping >2012-09-02 17:03:42,032 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-80) [154a4686] DbUserCacheManager::refreshAllUserData() - entered >2012-09-02 17:05:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-95) [4952ea3d] Autorecovering hosts is disabled, skipping >2012-09-02 17:05:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-95) [4952ea3d] Autorecovering storage domains is disabled, skipping >2012-09-02 17:10:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-27) [a89e0e7] Autorecovering hosts is disabled, skipping >2012-09-02 17:10:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-27) [a89e0e7] Autorecovering storage domains is disabled, skipping >2012-09-02 17:15:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-58) Autorecovering hosts is disabled, skipping >2012-09-02 17:15:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-58) Autorecovering storage domains is disabled, skipping >2012-09-02 17:20:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-81) [28d52053] Autorecovering hosts is disabled, skipping >2012-09-02 17:20:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-81) [28d52053] Autorecovering storage domains is disabled, skipping >2012-09-02 17:25:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-13) [3cc618be] Autorecovering hosts is disabled, skipping >2012-09-02 17:25:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-13) [3cc618be] Autorecovering storage domains is disabled, skipping >2012-09-02 17:30:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-39) [62249710] Autorecovering hosts is disabled, skipping >2012-09-02 17:30:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-39) [62249710] Autorecovering storage domains is disabled, skipping >2012-09-02 17:33:46,061 INFO [org.ovirt.engine.core.bll.LogoutUserCommand] (ajp-/127.0.0.1:8009-9) [12c44586] Running command: LogoutUserCommand internal: false. >2012-09-02 17:35:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-69) [39fd24f3] Autorecovering hosts is disabled, skipping >2012-09-02 17:35:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-69) [39fd24f3] Autorecovering storage domains is disabled, skipping >2012-09-02 17:40:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-97) [55a6af91] Autorecovering hosts is disabled, skipping >2012-09-02 17:40:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-97) [55a6af91] Autorecovering storage domains is disabled, skipping >2012-09-02 17:44:21,404 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8009-9) Checking if user admin@internal is an admin, result true >2012-09-02 17:44:21,405 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8009-9) Running command: LoginUserCommand internal: false. >2012-09-02 17:44:21,446 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8009-9) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2012-09-02 17:44:21,712 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8009-5) Checking if user admin@internal is an admin, result true >2012-09-02 17:44:21,712 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8009-5) Running command: LoginUserCommand internal: false. >2012-09-02 17:44:21,742 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8009-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2012-09-02 17:44:53,689 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-29) domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 17:45:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-37) [14f6fa8e] Autorecovering hosts is disabled, skipping >2012-09-02 17:45:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-37) [14f6fa8e] Autorecovering storage domains is disabled, skipping >2012-09-02 17:45:28,656 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-59) [4f91000b] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 17:45:30,664 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-79) ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 17:45:32,672 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-82) [f21968f] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 17:45:34,679 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-86) [1b72b674] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 17:45:36,400 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-89) [1b6b895d] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: bd560b80-c245-46b3-ad8c-b142a3460cf6 Type: StoragePool >2012-09-02 17:45:36,471 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-89) [1b6b895d] IrsBroker::Failed::GetStoragePoolInfoVDS due to: ConnectException: Connection refused >2012-09-02 17:45:36,686 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-92) ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 17:45:38,693 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-61) ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 17:45:40,700 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-88) ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 17:45:42,714 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-95) [4952ea3d] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 17:45:44,721 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-100) [6dae079f] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 17:45:46,480 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [55a6af91] IrsBroker::Failed::GetStoragePoolInfoVDS due to: ConnectException: Connection refused >2012-09-02 17:45:46,728 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-2) [2a8c60b0] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 17:45:48,735 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-5) [40eb8ded] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 17:45:50,743 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-99) [d9b82ea] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 17:45:52,763 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-7) ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 17:45:54,770 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-64) [5bdf963c] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 17:45:56,494 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-20) IrsBroker::Failed::GetStoragePoolInfoVDS due to: ConnectException: Connection refused >2012-09-02 17:45:56,778 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-12) [5064a4c0] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 17:45:58,790 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-16) [15c45473] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 17:46:00,796 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-14) [2ff7a817] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 17:46:02,803 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-19) [229837ce] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 17:46:04,810 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-25) ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 17:46:06,503 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-32) [65b08d21] IrsBroker::Failed::GetStoragePoolInfoVDS due to: ConnectException: Connection refused >2012-09-02 17:46:06,817 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-28) [164d9fbc] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 17:46:08,824 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-40) [31e89cb7] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 17:46:10,831 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-33) ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 17:46:12,838 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-43) [35748f38] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 17:46:14,845 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-36) [765c3ffe] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 17:46:16,511 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [1cf6d845] IrsBroker::Failed::GetStoragePoolInfoVDS due to: ConnectException: Connection refused >2012-09-02 17:46:16,852 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-46) ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 17:46:18,859 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-49) [767e356a] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 17:46:20,867 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-60) [dbe32c9] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 17:46:22,875 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-45) [4a22f461] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 17:46:24,882 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-38) [1a3821] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 17:46:26,520 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-54) IrsBroker::Failed::GetStoragePoolInfoVDS due to: ConnectException: Connection refused >2012-09-02 17:46:26,889 ERROR [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-50) [6dc69ea2] VDS::handleNetworkException Server failed to respond, vds_id = b32eacf0-f4e9-11e1-973c-001a4a169705, vds_name = green-vdsa.qa.lab.tlv.redhat.com, error = VDSNetworkException: >2012-09-02 17:46:26,912 INFO [org.ovirt.engine.core.bll.VdsEventListener] (pool-4-thread-47) [6dc69ea2] ResourceManager::vdsNotResponding entered for Host b32eacf0-f4e9-11e1-973c-001a4a169705, 10.35.102.10 >2012-09-02 17:46:26,941 WARN [org.ovirt.engine.core.bll.VdsNotRespondingTreatmentCommand] (pool-4-thread-47) [19bbee27] CanDoAction of action VdsNotRespondingTreatment failed. Reasons:VDS_FENCING_DISABLED >2012-09-02 17:46:35,808 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-91) [7b45712a] Running command: InitVdsOnUpCommand internal: true. >2012-09-02 17:46:35,875 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-91) [5c3b46c4] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: bd560b80-c245-46b3-ad8c-b142a3460cf6 Type: StoragePool >2012-09-02 17:46:35,877 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (QuartzScheduler_Worker-91) [5c3b46c4] START, ConnectStorageServerVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, storageType = ISCSI, connectionList = [{ id: 90cee3d3-f602-49f9-8f17-8665c22f2d06, connection: 10.35.64.10 };]), log id: 11ca3396 >2012-09-02 17:46:36,009 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (QuartzScheduler_Worker-91) [5c3b46c4] FINISH, ConnectStorageServerVDSCommand, return: {90cee3d3-f602-49f9-8f17-8665c22f2d06=0}, log id: 11ca3396 >2012-09-02 17:46:36,016 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-91) [5c3b46c4] Host green-vdsa.qa.lab.tlv.redhat.com storage connection was succeeded >2012-09-02 17:46:36,453 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-91) [5c3b46c4] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 264e0a02 >2012-09-02 17:46:37,345 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-91) [5c3b46c4] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 17:46:37,345 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-91) [5c3b46c4] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 17:46:37,345 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-91) [5c3b46c4] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:46:37,345 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-91) [5c3b46c4] FINISH, ConnectStoragePoolVDSCommand, log id: 264e0a02 >2012-09-02 17:46:37,345 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-91) [5c3b46c4] Could not connect host green-vdsa.qa.lab.tlv.redhat.com to pool TestDataCenter >2012-09-02 17:46:37,867 INFO [org.ovirt.engine.core.bll.MultipleActionsRunner] (pool-4-thread-47) [5c3b46c4] MultipleActionsRunner of type MigrateVmToServer invoked with no actions >2012-09-02 17:46:38,041 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-91) [29ec124e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: b32eacf0-f4e9-11e1-973c-001a4a169705 Type: VDS >2012-09-02 17:46:38,082 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-91) [1129d67] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: b32eacf0-f4e9-11e1-973c-001a4a169705 Type: VDS >2012-09-02 17:46:46,630 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [71797221] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 17:46:46,647 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-77) [71797221] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 1d1a6ceb >2012-09-02 17:46:47,511 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-77) [71797221] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 17:46:47,511 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-77) [71797221] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 17:46:47,511 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-77) [71797221] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:46:47,511 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-77) [71797221] FINISH, ConnectStoragePoolVDSCommand, log id: 1d1a6ceb >2012-09-02 17:46:47,511 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [71797221] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 17:46:47,511 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [71797221] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:46:48,112 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-77) [5aabc0e5] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 17:46:48,157 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-77) [5aabc0e5] START, ResetIrsVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = 10.35.102.10, vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, ignoreStopFailed = true), log id: 193c2313 >2012-09-02 17:46:48,158 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-77) [5aabc0e5] START, SpmStopVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6), log id: 1b849672 >2012-09-02 17:46:48,193 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-77) [5aabc0e5] SpmStopVDSCommand::Stopping SPM on vds green-vdsa.qa.lab.tlv.redhat.com, pool id bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 17:46:48,231 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-77) [5aabc0e5] FINISH, SpmStopVDSCommand, log id: 1b849672 >2012-09-02 17:46:48,248 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-77) [5aabc0e5] FINISH, ResetIrsVDSCommand, log id: 193c2313 >2012-09-02 17:46:48,258 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-77) [5aabc0e5] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 191b20a1 >2012-09-02 17:46:48,258 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [5aabc0e5] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 17:46:48,258 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-77) [5aabc0e5] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 191b20a1 >2012-09-02 17:46:48,314 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [5e430da1] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 17:46:58,347 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-4) [1b9aaac2] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 17:46:58,366 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-4) [1b9aaac2] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 4f16ff12 >2012-09-02 17:46:59,202 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-4) [1b9aaac2] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 17:46:59,202 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-4) [1b9aaac2] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 17:46:59,202 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-4) [1b9aaac2] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:46:59,202 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-4) [1b9aaac2] FINISH, ConnectStoragePoolVDSCommand, log id: 4f16ff12 >2012-09-02 17:46:59,202 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-4) [1b9aaac2] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 17:46:59,202 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-4) [1b9aaac2] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:46:59,322 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-4) [2eec22b4] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 17:46:59,390 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-4) [2eec22b4] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5632d3e >2012-09-02 17:46:59,390 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-4) [2eec22b4] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 17:46:59,390 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-4) [2eec22b4] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5632d3e >2012-09-02 17:47:09,319 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-9) [69982a55] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 17:47:09,445 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-18) [64c55214] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 17:47:09,457 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-18) [64c55214] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 17ea9e81 >2012-09-02 17:47:10,307 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-18) [64c55214] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 17:47:10,307 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-18) [64c55214] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 17:47:10,307 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-18) [64c55214] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:47:10,307 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-18) [64c55214] FINISH, ConnectStoragePoolVDSCommand, log id: 17ea9e81 >2012-09-02 17:47:10,307 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-18) [64c55214] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 17:47:10,307 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-18) [64c55214] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:47:10,700 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-18) [27b5e70] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 17:47:10,736 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-18) [27b5e70] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2300bb51 >2012-09-02 17:47:10,736 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-18) [27b5e70] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 17:47:10,736 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-18) [27b5e70] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2300bb51 >2012-09-02 17:47:19,523 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [592daca9] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 17:47:20,785 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-37) [14f6fa8e] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 17:47:20,809 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-37) [14f6fa8e] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 2b81c3e2 >2012-09-02 17:47:21,680 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-37) [14f6fa8e] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 17:47:21,680 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-37) [14f6fa8e] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 17:47:21,680 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-37) [14f6fa8e] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:47:21,680 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-37) [14f6fa8e] FINISH, ConnectStoragePoolVDSCommand, log id: 2b81c3e2 >2012-09-02 17:47:21,680 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-37) [14f6fa8e] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 17:47:21,680 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-37) [14f6fa8e] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:47:21,743 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-37) [794905fc] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 17:47:21,798 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-37) [794905fc] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6ef14fe2 >2012-09-02 17:47:21,798 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-37) [794905fc] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 17:47:21,798 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-37) [794905fc] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6ef14fe2 >2012-09-02 17:47:23,969 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-8) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=green-vdsa.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 17:47:30,433 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-56) domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 17:47:31,833 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-49) [767e356a] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 17:47:32,137 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-49) [767e356a] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 20420d4b >2012-09-02 17:47:32,997 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-49) [767e356a] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 17:47:32,997 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-49) [767e356a] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 17:47:32,997 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-49) [767e356a] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:47:32,997 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-49) [767e356a] FINISH, ConnectStoragePoolVDSCommand, log id: 20420d4b >2012-09-02 17:47:32,997 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-49) [767e356a] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 17:47:32,997 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-49) [767e356a] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:47:33,119 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-49) [2538bc32] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 17:47:33,154 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-49) [2538bc32] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1d9601a4 >2012-09-02 17:47:33,154 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-49) [2538bc32] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 17:47:33,154 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-49) [2538bc32] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1d9601a4 >2012-09-02 17:47:34,000 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-6) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=green-vdsa.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 17:47:41,122 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-58) domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 17:47:43,233 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [60860e0e] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 17:47:43,289 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-66) [60860e0e] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: a001d89 >2012-09-02 17:47:44,033 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-3) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=green-vdsa.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 17:47:44,171 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-66) [60860e0e] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 17:47:44,171 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-66) [60860e0e] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 17:47:44,172 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-66) [60860e0e] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:47:44,172 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-66) [60860e0e] FINISH, ConnectStoragePoolVDSCommand, log id: a001d89 >2012-09-02 17:47:44,172 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [60860e0e] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 17:47:44,172 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [60860e0e] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:47:44,270 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-66) [269bf226] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 17:47:44,312 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-66) [269bf226] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1571d74c >2012-09-02 17:47:44,312 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [269bf226] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 17:47:44,312 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-66) [269bf226] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1571d74c >2012-09-02 17:47:51,360 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-79) domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 17:47:54,075 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-7) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=green-vdsa.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 17:47:54,368 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) [1b72b674] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 17:47:54,386 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-86) [1b72b674] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 50ee7035 >2012-09-02 17:47:55,256 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-86) [1b72b674] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 17:47:55,256 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-86) [1b72b674] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 17:47:55,256 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-86) [1b72b674] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:47:55,256 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-86) [1b72b674] FINISH, ConnectStoragePoolVDSCommand, log id: 50ee7035 >2012-09-02 17:47:55,256 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) [1b72b674] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 17:47:55,256 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) [1b72b674] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:47:55,603 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-86) [4effdff3] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 17:47:55,751 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-86) [4effdff3] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 186b0be6 >2012-09-02 17:47:55,751 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) [4effdff3] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 17:47:55,751 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-86) [4effdff3] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 186b0be6 >2012-09-02 17:48:01,583 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-88) domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 17:48:04,104 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-11) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=green-vdsa.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 17:48:05,793 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [62351dda] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 17:48:06,653 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-87) [62351dda] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 4355448e >2012-09-02 17:48:07,526 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-87) [62351dda] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 17:48:07,526 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-87) [62351dda] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 17:48:07,526 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-87) [62351dda] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:48:07,526 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-87) [62351dda] FINISH, ConnectStoragePoolVDSCommand, log id: 4355448e >2012-09-02 17:48:07,526 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [62351dda] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 17:48:07,526 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [62351dda] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:48:07,677 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-87) [49e9877f] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 17:48:07,709 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-87) [49e9877f] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 68f3a10c >2012-09-02 17:48:07,709 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [49e9877f] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 17:48:07,709 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-87) [49e9877f] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 68f3a10c >2012-09-02 17:48:11,872 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [40eb8ded] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 17:48:14,144 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-2) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=green-vdsa.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 17:48:17,754 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-11) [13caede9] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 17:48:17,788 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-11) [13caede9] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 2a010552 >2012-09-02 17:48:18,650 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-11) [13caede9] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 17:48:18,650 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-11) [13caede9] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 17:48:18,650 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-11) [13caede9] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:48:18,650 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-11) [13caede9] FINISH, ConnectStoragePoolVDSCommand, log id: 2a010552 >2012-09-02 17:48:18,650 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-11) [13caede9] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 17:48:18,650 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-11) [13caede9] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:48:18,754 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-11) [28bf429a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 17:48:18,850 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-11) [28bf429a] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 42bc0a1 >2012-09-02 17:48:18,850 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-11) [28bf429a] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 17:48:18,850 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-11) [28bf429a] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 42bc0a1 >2012-09-02 17:48:22,159 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-15) [5147a4f8] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 17:48:24,187 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-5) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=green-vdsa.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 17:48:28,917 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-24) [392574c] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 17:48:29,165 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-24) [392574c] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: b37a72e >2012-09-02 17:48:30,023 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-24) [392574c] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 17:48:30,023 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-24) [392574c] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 17:48:30,023 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-24) [392574c] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:48:30,023 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-24) [392574c] FINISH, ConnectStoragePoolVDSCommand, log id: b37a72e >2012-09-02 17:48:30,024 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-24) [392574c] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 17:48:30,024 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-24) [392574c] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:48:31,048 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-24) [699cf9f8] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 17:48:31,117 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-24) [699cf9f8] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 233ce89e >2012-09-02 17:48:31,117 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-24) [699cf9f8] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 17:48:31,117 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-24) [699cf9f8] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 233ce89e >2012-09-02 17:48:33,195 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-33) domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 17:48:34,212 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-8) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=green-vdsa.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 17:48:41,165 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-47) [7061330f] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 17:48:41,209 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-47) [7061330f] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 37895a91 >2012-09-02 17:48:42,062 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-47) [7061330f] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 17:48:42,062 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-47) [7061330f] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 17:48:42,062 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-47) [7061330f] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:48:42,062 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-47) [7061330f] FINISH, ConnectStoragePoolVDSCommand, log id: 37895a91 >2012-09-02 17:48:42,062 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-47) [7061330f] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 17:48:42,062 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-47) [7061330f] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:48:42,148 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-47) [702f63e4] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 17:48:42,199 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-47) [702f63e4] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 116e805 >2012-09-02 17:48:42,199 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-47) [702f63e4] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 17:48:42,199 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-47) [702f63e4] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 116e805 >2012-09-02 17:48:43,406 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-52) [570552a] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 17:48:44,247 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-6) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=green-vdsa.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 17:48:52,258 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) [1bbf829c] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 17:48:52,283 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-55) [1bbf829c] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 5d709815 >2012-09-02 17:48:53,141 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-55) [1bbf829c] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 17:48:53,141 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-55) [1bbf829c] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 17:48:53,141 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-55) [1bbf829c] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:48:53,141 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-55) [1bbf829c] FINISH, ConnectStoragePoolVDSCommand, log id: 5d709815 >2012-09-02 17:48:53,142 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) [1bbf829c] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 17:48:53,142 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) [1bbf829c] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:48:53,214 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-55) [65087b1b] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 17:48:53,251 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-55) [65087b1b] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 39b9b505 >2012-09-02 17:48:53,251 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) [65087b1b] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 17:48:53,251 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-55) [65087b1b] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 39b9b505 >2012-09-02 17:48:53,620 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [6af76567] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 17:48:54,282 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-3) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=green-vdsa.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 17:49:03,335 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [f21968f] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 17:49:03,356 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-82) [f21968f] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 415571c >2012-09-02 17:49:04,210 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-82) [f21968f] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 17:49:04,210 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-82) [f21968f] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 17:49:04,210 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-82) [f21968f] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:49:04,210 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-82) [f21968f] FINISH, ConnectStoragePoolVDSCommand, log id: 415571c >2012-09-02 17:49:04,210 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [f21968f] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 17:49:04,210 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [f21968f] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:49:04,285 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-82) [32a0233d] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 17:49:04,320 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-7) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=green-vdsa.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 17:49:04,326 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-82) [32a0233d] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 41e9f53a >2012-09-02 17:49:04,326 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [32a0233d] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 17:49:04,326 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-82) [32a0233d] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 41e9f53a >2012-09-02 17:49:14,050 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [f848894] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 17:49:14,360 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-4) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=green-vdsa.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 17:49:14,371 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-93) [6cfa5fec] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 17:49:14,387 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-93) [6cfa5fec] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 182486cd >2012-09-02 17:49:15,254 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-93) [6cfa5fec] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 17:49:15,254 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-93) [6cfa5fec] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 17:49:15,254 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-93) [6cfa5fec] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:49:15,254 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-93) [6cfa5fec] FINISH, ConnectStoragePoolVDSCommand, log id: 182486cd >2012-09-02 17:49:15,254 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-93) [6cfa5fec] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 17:49:15,254 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-93) [6cfa5fec] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:49:15,522 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-93) [361e27ee] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 17:49:15,567 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-93) [361e27ee] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 56f2b33c >2012-09-02 17:49:15,568 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-93) [361e27ee] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 17:49:15,568 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-93) [361e27ee] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 56f2b33c >2012-09-02 17:49:24,266 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [1f04073e] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 17:49:24,395 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-2) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=green-vdsa.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 17:49:25,621 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) [29507acf] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 17:49:25,638 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-10) [29507acf] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 72aaa585 >2012-09-02 17:49:26,497 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-10) [29507acf] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 17:49:26,497 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-10) [29507acf] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 17:49:26,497 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-10) [29507acf] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:49:26,497 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-10) [29507acf] FINISH, ConnectStoragePoolVDSCommand, log id: 72aaa585 >2012-09-02 17:49:26,497 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) [29507acf] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 17:49:26,497 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) [29507acf] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:49:27,526 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-10) [48a15dfc] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 17:49:27,657 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-10) [48a15dfc] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2dc86abd >2012-09-02 17:49:27,657 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) [48a15dfc] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 17:49:27,657 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-10) [48a15dfc] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2dc86abd >2012-09-02 17:49:34,436 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-5) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=green-vdsa.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 17:49:34,811 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-29) domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 17:49:37,950 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) [4b5ea5e5] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 17:49:38,269 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-21) [4b5ea5e5] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 5b7d2f38 >2012-09-02 17:49:39,128 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-21) [4b5ea5e5] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 17:49:39,128 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-21) [4b5ea5e5] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 17:49:39,128 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-21) [4b5ea5e5] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:49:39,128 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-21) [4b5ea5e5] FINISH, ConnectStoragePoolVDSCommand, log id: 5b7d2f38 >2012-09-02 17:49:39,128 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) [4b5ea5e5] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 17:49:39,128 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) [4b5ea5e5] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:49:39,645 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-21) [2eb7fc9d] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 17:49:39,678 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-21) [2eb7fc9d] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: d0823ae >2012-09-02 17:49:39,678 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) [2eb7fc9d] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 17:49:39,678 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-21) [2eb7fc9d] FINISH, MarkPoolInReconstructModeVDSCommand, log id: d0823ae >2012-09-02 17:49:44,475 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-8) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=green-vdsa.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 17:49:45,386 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [4722d115] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 17:49:49,713 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-47) [702f63e4] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 17:49:49,762 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-47) [702f63e4] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 595079e1 >2012-09-02 17:49:50,621 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-47) [702f63e4] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 17:49:50,621 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-47) [702f63e4] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 17:49:50,621 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-47) [702f63e4] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:49:50,621 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-47) [702f63e4] FINISH, ConnectStoragePoolVDSCommand, log id: 595079e1 >2012-09-02 17:49:50,621 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-47) [702f63e4] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 17:49:50,621 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-47) [702f63e4] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:49:50,690 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-47) [1bf6f4f5] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 17:49:50,724 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-47) [1bf6f4f5] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3f8784ec >2012-09-02 17:49:50,724 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-47) [1bf6f4f5] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 17:49:50,724 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-47) [1bf6f4f5] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3f8784ec >2012-09-02 17:49:54,513 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-6) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=green-vdsa.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 17:49:55,740 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) [46ff88ea] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 17:50:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-68) Autorecovering hosts is disabled, skipping >2012-09-02 17:50:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-68) Autorecovering storage domains is disabled, skipping >2012-09-02 17:50:00,775 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) [ffaf13b] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 17:50:01,253 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-65) [ffaf13b] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: c4f4177 >2012-09-02 17:50:02,128 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-65) [ffaf13b] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 17:50:02,128 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-65) [ffaf13b] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 17:50:02,128 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-65) [ffaf13b] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:50:02,128 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-65) [ffaf13b] FINISH, ConnectStoragePoolVDSCommand, log id: c4f4177 >2012-09-02 17:50:02,128 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) [ffaf13b] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 17:50:02,128 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) [ffaf13b] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:50:03,365 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-65) [1fcd574e] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 17:50:03,409 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-65) [1fcd574e] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1c52a4f0 >2012-09-02 17:50:03,409 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) [1fcd574e] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 17:50:03,409 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-65) [1fcd574e] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1c52a4f0 >2012-09-02 17:50:04,554 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-3) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=green-vdsa.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 17:50:05,964 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-75) [1b3d0324] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 17:50:13,452 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [32a0233d] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 17:50:13,472 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-82) [32a0233d] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 46d88a88 >2012-09-02 17:50:14,371 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-82) [32a0233d] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 17:50:14,371 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-82) [32a0233d] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 17:50:14,371 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-82) [32a0233d] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:50:14,371 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-82) [32a0233d] FINISH, ConnectStoragePoolVDSCommand, log id: 46d88a88 >2012-09-02 17:50:14,371 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [32a0233d] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 17:50:14,371 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [32a0233d] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:50:14,480 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-82) [77190e4a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 17:50:14,517 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-82) [77190e4a] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1f4ae7bd >2012-09-02 17:50:14,517 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [77190e4a] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 17:50:14,517 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-82) [77190e4a] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1f4ae7bd >2012-09-02 17:50:14,591 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-9) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=green-vdsa.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 17:50:16,201 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-92) domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 17:50:24,548 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [59ece3a8] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 17:50:24,594 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-94) [59ece3a8] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 1490b9d7 >2012-09-02 17:50:24,627 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-4) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=green-vdsa.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 17:50:25,449 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-94) [59ece3a8] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 17:50:25,449 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-94) [59ece3a8] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 17:50:25,449 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-94) [59ece3a8] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:50:25,449 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-94) [59ece3a8] FINISH, ConnectStoragePoolVDSCommand, log id: 1490b9d7 >2012-09-02 17:50:25,449 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [59ece3a8] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 17:50:25,449 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [59ece3a8] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:50:25,692 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-94) [cbe470] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 17:50:25,727 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-94) [cbe470] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 69b84535 >2012-09-02 17:50:25,727 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [cbe470] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 17:50:25,727 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-94) [cbe470] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 69b84535 >2012-09-02 17:50:26,934 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [49e9877f] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 17:50:34,655 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-2) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=green-vdsa.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 17:50:35,766 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-20) hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 17:50:35,783 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-20) START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 5419d5c >2012-09-02 17:50:36,641 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-20) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 17:50:36,641 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-20) Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 17:50:36,642 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-20) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:50:36,642 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-20) FINISH, ConnectStoragePoolVDSCommand, log id: 5419d5c >2012-09-02 17:50:36,642 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-20) IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 17:50:36,642 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-20) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:50:36,718 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-20) [330b8136] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 17:50:36,762 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-20) [330b8136] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 41a0580 >2012-09-02 17:50:36,762 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-20) [330b8136] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 17:50:36,762 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-20) [330b8136] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 41a0580 >2012-09-02 17:50:37,144 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-12) [5064a4c0] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 17:50:44,690 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-5) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=green-vdsa.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 17:50:46,803 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [592daca9] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 17:50:46,816 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-23) [592daca9] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 15a9bf53 >2012-09-02 17:50:47,642 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-23) [592daca9] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 17:50:47,642 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-23) [592daca9] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 17:50:47,642 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-23) [592daca9] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:50:47,642 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-23) [592daca9] FINISH, ConnectStoragePoolVDSCommand, log id: 15a9bf53 >2012-09-02 17:50:47,642 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [592daca9] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 17:50:47,642 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [592daca9] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:50:48,117 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-23) [7648219e] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 17:50:48,194 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-23) [7648219e] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 78d836a8 >2012-09-02 17:50:48,194 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [7648219e] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 17:50:48,194 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-23) [7648219e] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 78d836a8 >2012-09-02 17:50:54,725 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-8) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=green-vdsa.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 17:50:57,571 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [1cf6d845] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 17:50:58,229 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 17:50:58,425 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-46) START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 702aff64 >2012-09-02 17:50:59,278 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 17:50:59,278 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-46) Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 17:50:59,278 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:50:59,278 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-46) FINISH, ConnectStoragePoolVDSCommand, log id: 702aff64 >2012-09-02 17:50:59,279 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 17:50:59,279 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:50:59,353 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-46) [69d4155f] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 17:50:59,388 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-46) [69d4155f] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 536de3d7 >2012-09-02 17:50:59,388 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [69d4155f] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 17:50:59,388 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-46) [69d4155f] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 536de3d7 >2012-09-02 17:51:04,755 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-6) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=green-vdsa.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 17:51:07,852 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-54) domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 17:51:09,453 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-58) hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 17:51:10,104 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-58) START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 2d042920 >2012-09-02 17:51:10,977 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-58) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 17:51:10,977 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-58) Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 17:51:10,977 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-58) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:51:10,977 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-58) FINISH, ConnectStoragePoolVDSCommand, log id: 2d042920 >2012-09-02 17:51:10,977 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-58) IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 17:51:10,977 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-58) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:51:11,165 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-58) [7d786bb3] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 17:51:11,201 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-58) [7d786bb3] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 632c673c >2012-09-02 17:51:11,201 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-58) [7d786bb3] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 17:51:11,201 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-58) [7d786bb3] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 632c673c >2012-09-02 17:51:14,798 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-10) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=green-vdsa.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 17:51:18,060 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [12d00b82] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 17:51:21,237 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [7a07ce1a] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 17:51:21,253 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-71) [7a07ce1a] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 316cbfe0 >2012-09-02 17:51:22,111 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-71) [7a07ce1a] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 17:51:22,112 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-71) [7a07ce1a] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 17:51:22,112 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-71) [7a07ce1a] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:51:22,112 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-71) [7a07ce1a] FINISH, ConnectStoragePoolVDSCommand, log id: 316cbfe0 >2012-09-02 17:51:22,112 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [7a07ce1a] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 17:51:22,112 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [7a07ce1a] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:51:22,194 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-71) [ba9ffb2] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 17:51:22,221 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-71) [ba9ffb2] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3eb9610a >2012-09-02 17:51:22,221 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [ba9ffb2] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 17:51:22,221 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-71) [ba9ffb2] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3eb9610a >2012-09-02 17:51:24,833 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-9) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=green-vdsa.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 17:51:28,364 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 17:51:32,263 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [239c5a4] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 17:51:32,286 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-96) [239c5a4] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 29d17a4d >2012-09-02 17:51:33,145 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-96) [239c5a4] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 17:51:33,145 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-96) [239c5a4] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 17:51:33,145 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-96) [239c5a4] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:51:33,145 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-96) [239c5a4] FINISH, ConnectStoragePoolVDSCommand, log id: 29d17a4d >2012-09-02 17:51:33,145 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [239c5a4] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 17:51:33,145 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [239c5a4] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:51:33,421 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-96) [361daa41] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 17:51:33,465 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-96) [361daa41] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: cfe3b58 >2012-09-02 17:51:33,465 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [361daa41] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 17:51:33,465 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-96) [361daa41] FINISH, MarkPoolInReconstructModeVDSCommand, log id: cfe3b58 >2012-09-02 17:51:34,868 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-4) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=green-vdsa.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 17:51:38,958 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) [2a8c60b0] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 17:51:43,505 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) [3cc618be] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 17:51:43,544 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-13) [3cc618be] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 7ef16b8b >2012-09-02 17:51:44,408 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-13) [3cc618be] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 17:51:44,408 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-13) [3cc618be] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 17:51:44,408 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-13) [3cc618be] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:51:44,408 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-13) [3cc618be] FINISH, ConnectStoragePoolVDSCommand, log id: 7ef16b8b >2012-09-02 17:51:44,408 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) [3cc618be] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 17:51:44,408 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) [3cc618be] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:51:44,530 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-13) [6709ab6a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 17:51:44,564 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-13) [6709ab6a] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 32eced0c >2012-09-02 17:51:44,564 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) [6709ab6a] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 17:51:44,564 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-13) [6709ab6a] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 32eced0c >2012-09-02 17:51:44,904 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-2) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=green-vdsa.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 17:51:49,207 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-15) [5147a4f8] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 17:51:54,601 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 17:51:54,620 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-25) START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 6910d8f8 >2012-09-02 17:51:54,933 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-5) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=green-vdsa.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 17:51:55,494 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-25) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 17:51:55,494 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-25) Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 17:51:55,494 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-25) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:51:55,494 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-25) FINISH, ConnectStoragePoolVDSCommand, log id: 6910d8f8 >2012-09-02 17:51:55,494 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 17:51:55,494 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:51:55,599 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-25) [45a8613b] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 17:51:55,669 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-25) [45a8613b] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7fffc437 >2012-09-02 17:51:55,670 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [45a8613b] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 17:51:55,670 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-25) [45a8613b] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7fffc437 >2012-09-02 17:51:59,435 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-37) [794905fc] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 17:52:04,969 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-8) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=green-vdsa.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 17:52:05,704 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [62249710] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 17:52:06,168 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-39) [62249710] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 1a674007 >2012-09-02 17:52:07,025 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-39) [62249710] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 17:52:07,025 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-39) [62249710] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 17:52:07,025 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-39) [62249710] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:52:07,026 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-39) [62249710] FINISH, ConnectStoragePoolVDSCommand, log id: 1a674007 >2012-09-02 17:52:07,026 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [62249710] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 17:52:07,026 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [62249710] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:52:07,109 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-39) [6c3830ce] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 17:52:07,143 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-39) [6c3830ce] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 55d0ec5e >2012-09-02 17:52:07,143 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [6c3830ce] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 17:52:07,143 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-39) [6c3830ce] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 55d0ec5e >2012-09-02 17:52:09,874 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) [dbe32c9] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 17:52:14,993 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-11) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=green-vdsa.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 17:52:17,193 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [3b94000c] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 17:52:17,249 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-73) [3b94000c] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 2888707c >2012-09-02 17:52:18,110 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-73) [3b94000c] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 17:52:18,110 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-73) [3b94000c] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 17:52:18,110 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-73) [3b94000c] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:52:18,110 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-73) [3b94000c] FINISH, ConnectStoragePoolVDSCommand, log id: 2888707c >2012-09-02 17:52:18,110 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [3b94000c] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 17:52:18,110 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [3b94000c] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:52:18,581 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-73) [3ce1db36] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 17:52:18,636 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-73) [3ce1db36] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3413f122 >2012-09-02 17:52:18,636 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [3ce1db36] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 17:52:18,636 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-73) [3ce1db36] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3413f122 >2012-09-02 17:52:20,434 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [6af76567] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 17:52:25,027 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-10) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=green-vdsa.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 17:52:28,718 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-80) [154a4686] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 17:52:29,089 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-80) [154a4686] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 36eb0d83 >2012-09-02 17:52:30,182 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-80) [154a4686] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 17:52:30,182 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-80) [154a4686] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 17:52:30,182 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-80) [154a4686] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:52:30,182 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-80) [154a4686] FINISH, ConnectStoragePoolVDSCommand, log id: 36eb0d83 >2012-09-02 17:52:30,182 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-80) [154a4686] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 17:52:30,182 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-80) [154a4686] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:52:30,434 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [ba9ffb2] starting ProcessDomainRecovery for domain 56a64ce1-90ab-4576-86fa-c269160aaa38 >2012-09-02 17:52:30,445 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [ba9ffb2] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 was reported by all hosts in status UP as problematic. But not moving the Domain to NonOperational. Because of is reconstract now >2012-09-02 17:52:31,034 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-80) [655605ac] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 17:52:31,071 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-80) [655605ac] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 469d105b >2012-09-02 17:52:31,071 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-80) [655605ac] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 17:52:31,071 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-80) [655605ac] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 469d105b >2012-09-02 17:52:35,062 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-9) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=green-vdsa.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 17:52:41,118 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [361daa41] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 17:52:41,141 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-96) [361daa41] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 2708038c >2012-09-02 17:52:41,266 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-100) [6dae079f] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 17:52:42,140 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-96) [361daa41] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 17:52:42,140 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-96) [361daa41] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 17:52:42,140 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-96) [361daa41] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:52:42,140 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-96) [361daa41] FINISH, ConnectStoragePoolVDSCommand, log id: 2708038c >2012-09-02 17:52:42,140 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [361daa41] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 17:52:42,140 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [361daa41] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:52:42,558 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-96) [11d202] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 17:52:42,601 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-96) [11d202] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 468e2190 >2012-09-02 17:52:42,601 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [11d202] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 17:52:42,601 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-96) [11d202] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 468e2190 >2012-09-02 17:52:45,092 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-4) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=green-vdsa.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 17:52:51,507 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) [6709ab6a] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 17:52:52,636 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) [48a15dfc] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 17:52:52,660 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-10) [48a15dfc] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 4080e8b >2012-09-02 17:52:53,539 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-10) [48a15dfc] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 17:52:53,539 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-10) [48a15dfc] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 17:52:53,539 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-10) [48a15dfc] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:52:53,539 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-10) [48a15dfc] FINISH, ConnectStoragePoolVDSCommand, log id: 4080e8b >2012-09-02 17:52:53,539 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) [48a15dfc] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 17:52:53,539 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) [48a15dfc] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:52:53,608 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-10) [7fa2eb8b] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 17:52:53,641 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-10) [7fa2eb8b] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 174af205 >2012-09-02 17:52:53,641 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) [7fa2eb8b] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 17:52:53,641 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-10) [7fa2eb8b] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 174af205 >2012-09-02 17:52:55,120 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-2) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=green-vdsa.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 17:53:01,755 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-32) [65b08d21] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 17:53:03,676 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [45a8613b] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 17:53:04,504 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-25) [45a8613b] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 6b2d86c6 >2012-09-02 17:53:05,156 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-5) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=green-vdsa.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 17:53:05,367 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-25) [45a8613b] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 17:53:05,367 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-25) [45a8613b] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 17:53:05,367 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-25) [45a8613b] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:53:05,367 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-25) [45a8613b] FINISH, ConnectStoragePoolVDSCommand, log id: 6b2d86c6 >2012-09-02 17:53:05,367 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [45a8613b] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 17:53:05,367 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [45a8613b] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:53:05,519 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-25) [24ffe179] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 17:53:05,562 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-25) [24ffe179] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6f5c6e7a >2012-09-02 17:53:05,562 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [24ffe179] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 17:53:05,562 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-25) [24ffe179] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6f5c6e7a >2012-09-02 17:53:12,021 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-36) [765c3ffe] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 17:53:15,193 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-7) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=green-vdsa.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 17:53:15,605 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-47) [1bf6f4f5] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 17:53:15,619 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-47) [1bf6f4f5] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 12a4a91e >2012-09-02 17:53:16,482 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-47) [1bf6f4f5] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 17:53:16,482 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-47) [1bf6f4f5] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 17:53:16,482 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-47) [1bf6f4f5] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:53:16,482 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-47) [1bf6f4f5] FINISH, ConnectStoragePoolVDSCommand, log id: 12a4a91e >2012-09-02 17:53:16,482 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-47) [1bf6f4f5] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 17:53:16,483 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-47) [1bf6f4f5] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:53:16,573 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-47) [1c260ac3] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 17:53:16,682 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-47) [1c260ac3] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 74574531 >2012-09-02 17:53:16,683 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-47) [1c260ac3] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 17:53:16,683 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-47) [1c260ac3] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 74574531 >2012-09-02 17:53:22,223 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-62) [3ea43f70] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 17:53:25,223 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-11) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=green-vdsa.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 17:53:26,717 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) [1fcd574e] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 17:53:26,999 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-65) [1fcd574e] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 41b30bd0 >2012-09-02 17:53:27,864 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-65) [1fcd574e] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 17:53:27,864 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-65) [1fcd574e] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 17:53:27,864 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-65) [1fcd574e] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:53:27,864 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-65) [1fcd574e] FINISH, ConnectStoragePoolVDSCommand, log id: 41b30bd0 >2012-09-02 17:53:27,864 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) [1fcd574e] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 17:53:27,864 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) [1fcd574e] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:53:27,939 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-65) [5dc6965d] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 17:53:27,984 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-65) [5dc6965d] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1caf1710 >2012-09-02 17:53:27,984 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) [5dc6965d] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 17:53:27,984 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-65) [5dc6965d] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1caf1710 >2012-09-02 17:53:32,462 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-91) [1129d67] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 17:53:33,196 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-70) [4679b9c1] starting ProcessDomainRecovery for domain 56a64ce1-90ab-4576-86fa-c269160aaa38 >2012-09-02 17:53:33,302 WARN [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (QuartzScheduler_Worker-70) [44df6371] The message key HandleFailedStorageDomain is missing from bundles/ExecutionMessages >2012-09-02 17:53:33,374 INFO [org.ovirt.engine.core.bll.storage.HandleFailedStorageDomainCommand] (QuartzScheduler_Worker-70) [44df6371] Running command: HandleFailedStorageDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 17:53:33,388 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-70) [413d2ba7] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 17:53:33,518 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (QuartzScheduler_Worker-70) [413d2ba7] START, DisconnectStorageServerVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, storageType = ISCSI, connectionList = [{ id: 90cee3d3-f602-49f9-8f17-8665c22f2d06, connection: 10.35.64.10 };]), log id: 1783da9e >2012-09-02 17:53:34,165 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (QuartzScheduler_Worker-70) [413d2ba7] FINISH, DisconnectStorageServerVDSCommand, return: {90cee3d3-f602-49f9-8f17-8665c22f2d06=0}, log id: 1783da9e >2012-09-02 17:53:34,166 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-70) [413d2ba7] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 726047aa >2012-09-02 17:53:34,166 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-70) [413d2ba7] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 17:53:34,166 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-70) [413d2ba7] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 726047aa >2012-09-02 17:53:34,175 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-70) [413d2ba7] No string for UNASSIGNED type. Use default Log >2012-09-02 17:53:35,258 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-10) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=green-vdsa.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 17:53:38,019 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-80) [655605ac] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 17:53:38,052 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-80) [655605ac] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 40310995 >2012-09-02 17:53:38,947 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-80) [655605ac] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 17:53:38,947 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-80) [655605ac] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 17:53:38,947 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-80) [655605ac] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:53:38,947 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-80) [655605ac] FINISH, ConnectStoragePoolVDSCommand, log id: 40310995 >2012-09-02 17:53:38,947 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-80) [655605ac] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 17:53:38,947 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-80) [655605ac] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:53:39,044 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-80) [5a9aa0c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 17:53:39,093 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-80) [5a9aa0c] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1f547a6a >2012-09-02 17:53:39,093 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-80) [5a9aa0c] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 17:53:39,093 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-80) [5a9aa0c] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1f547a6a >2012-09-02 17:53:45,293 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-9) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=green-vdsa.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 17:53:49,136 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-6) hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 17:53:49,161 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-6) START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 6dfdeffa >2012-09-02 17:53:50,048 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-6) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 17:53:50,048 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-6) Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 17:53:50,048 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-6) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:53:50,048 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-6) FINISH, ConnectStoragePoolVDSCommand, log id: 6dfdeffa >2012-09-02 17:53:50,048 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-6) IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 17:53:50,048 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-6) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:53:50,314 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-6) [7571ba07] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 17:53:50,492 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-6) [7571ba07] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5039fe27 >2012-09-02 17:53:50,492 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-6) [7571ba07] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 17:53:50,492 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-6) [7571ba07] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5039fe27 >2012-09-02 17:53:55,321 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-4) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=green-vdsa.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 17:54:00,548 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [15c45473] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 17:54:00,558 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-16) [15c45473] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 56acbd89 >2012-09-02 17:54:01,415 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-16) [15c45473] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 17:54:01,415 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-16) [15c45473] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 17:54:01,416 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-16) [15c45473] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:54:01,416 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-16) [15c45473] FINISH, ConnectStoragePoolVDSCommand, log id: 56acbd89 >2012-09-02 17:54:01,416 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [15c45473] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 17:54:01,416 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [15c45473] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:54:01,508 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-16) [22836716] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 17:54:01,537 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-16) [22836716] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2bf52f72 >2012-09-02 17:54:01,537 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [22836716] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 17:54:01,537 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-16) [22836716] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2bf52f72 >2012-09-02 17:54:05,357 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-2) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=green-vdsa.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 17:54:11,576 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-24) [699cf9f8] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 17:54:11,600 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-24) [699cf9f8] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 73de7c73 >2012-09-02 17:54:12,450 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-24) [699cf9f8] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 17:54:12,451 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-24) [699cf9f8] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 17:54:12,451 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-24) [699cf9f8] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:54:12,451 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-24) [699cf9f8] FINISH, ConnectStoragePoolVDSCommand, log id: 73de7c73 >2012-09-02 17:54:12,451 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-24) [699cf9f8] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 17:54:12,451 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-24) [699cf9f8] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:54:12,952 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-24) [31e52f20] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 17:54:12,982 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-24) [31e52f20] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4b1277c7 >2012-09-02 17:54:12,982 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-24) [31e52f20] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 17:54:12,982 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-24) [31e52f20] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4b1277c7 >2012-09-02 17:54:15,397 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-3) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=green-vdsa.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 17:54:23,031 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-47) [1c260ac3] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 17:54:23,071 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-47) [1c260ac3] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 61355490 >2012-09-02 17:54:23,930 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-47) [1c260ac3] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 17:54:23,930 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-47) [1c260ac3] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 17:54:23,930 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-47) [1c260ac3] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:54:23,930 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-47) [1c260ac3] FINISH, ConnectStoragePoolVDSCommand, log id: 61355490 >2012-09-02 17:54:23,930 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-47) [1c260ac3] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 17:54:23,930 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-47) [1c260ac3] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:54:24,070 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-47) [9d34e64] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 17:54:24,104 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-47) [9d34e64] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6a6f1a5a >2012-09-02 17:54:24,104 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-47) [9d34e64] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 17:54:24,104 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-47) [9d34e64] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6a6f1a5a >2012-09-02 17:54:25,433 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-7) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=green-vdsa.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 17:54:34,153 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-22) [4d76203] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 17:54:34,181 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-22) [4d76203] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 5e1a1ceb >2012-09-02 17:54:35,039 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-22) [4d76203] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 17:54:35,039 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-22) [4d76203] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 17:54:35,039 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-22) [4d76203] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:54:35,039 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-22) [4d76203] FINISH, ConnectStoragePoolVDSCommand, log id: 5e1a1ceb >2012-09-02 17:54:35,039 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-22) [4d76203] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 17:54:35,039 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-22) [4d76203] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:54:35,394 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-22) [632ea3f6] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 17:54:35,425 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-22) [632ea3f6] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 29c765d >2012-09-02 17:54:35,425 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-22) [632ea3f6] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 17:54:35,425 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-22) [632ea3f6] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 29c765d >2012-09-02 17:54:35,465 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-11) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=green-vdsa.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 17:54:45,460 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 17:54:45,501 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-10) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=green-vdsa.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 17:54:45,516 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-61) START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 2fde0ced >2012-09-02 17:54:46,372 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-61) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 17:54:46,372 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-61) Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 17:54:46,372 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-61) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:54:46,372 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-61) FINISH, ConnectStoragePoolVDSCommand, log id: 2fde0ced >2012-09-02 17:54:46,372 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 17:54:46,372 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:54:46,441 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-61) [7d8f5d69] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 17:54:46,493 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-61) [7d8f5d69] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: ab87258 >2012-09-02 17:54:46,493 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [7d8f5d69] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 17:54:46,493 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-61) [7d8f5d69] FINISH, MarkPoolInReconstructModeVDSCommand, log id: ab87258 >2012-09-02 17:54:55,538 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-9) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=green-vdsa.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 17:54:56,540 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-6) [7571ba07] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 17:54:56,560 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-6) [7571ba07] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 427b1fa4 >2012-09-02 17:54:57,433 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-6) [7571ba07] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 17:54:57,433 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-6) [7571ba07] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 17:54:57,433 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-6) [7571ba07] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:54:57,433 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-6) [7571ba07] FINISH, ConnectStoragePoolVDSCommand, log id: 427b1fa4 >2012-09-02 17:54:57,433 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-6) [7571ba07] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 17:54:57,433 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-6) [7571ba07] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:54:57,524 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-6) [1dd5be77] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 17:54:57,559 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-6) [1dd5be77] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 59abc38e >2012-09-02 17:54:57,559 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-6) [1dd5be77] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 17:54:57,559 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-6) [1dd5be77] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 59abc38e >2012-09-02 17:55:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-7) Autorecovering hosts is disabled, skipping >2012-09-02 17:55:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-7) Autorecovering storage domains is disabled, skipping >2012-09-02 17:55:05,574 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-4) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=green-vdsa.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 17:55:07,607 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [2ff7a817] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 17:55:07,656 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-14) [2ff7a817] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 6516c10f >2012-09-02 17:55:08,570 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-14) [2ff7a817] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 17:55:08,570 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-14) [2ff7a817] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 17:55:08,570 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-14) [2ff7a817] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:55:08,570 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-14) [2ff7a817] FINISH, ConnectStoragePoolVDSCommand, log id: 6516c10f >2012-09-02 17:55:08,570 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [2ff7a817] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 17:55:08,570 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [2ff7a817] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:55:09,022 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-14) [47f4fd8c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 17:55:09,055 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-14) [47f4fd8c] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 8091206 >2012-09-02 17:55:09,055 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [47f4fd8c] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 17:55:09,055 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-14) [47f4fd8c] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 8091206 >2012-09-02 17:55:15,609 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-6) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=green-vdsa.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 17:55:19,105 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [4722d115] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 17:55:19,180 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-30) [4722d115] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 14e29538 >2012-09-02 17:55:20,064 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-30) [4722d115] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 17:55:20,064 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-30) [4722d115] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 17:55:20,064 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-30) [4722d115] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:55:20,064 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-30) [4722d115] FINISH, ConnectStoragePoolVDSCommand, log id: 14e29538 >2012-09-02 17:55:20,064 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [4722d115] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 17:55:20,064 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [4722d115] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:55:20,564 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-30) [5a2190ec] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 17:55:20,650 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-30) [5a2190ec] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2b195a66 >2012-09-02 17:55:20,650 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [5a2190ec] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 17:55:20,650 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-30) [5a2190ec] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2b195a66 >2012-09-02 17:55:25,648 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-3) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=green-vdsa.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 17:55:30,697 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-34) [39d95b35] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 17:55:31,092 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-34) [39d95b35] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 35aff7e0 >2012-09-02 17:55:31,963 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-34) [39d95b35] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 17:55:31,964 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-34) [39d95b35] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 17:55:31,964 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-34) [39d95b35] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:55:31,964 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-34) [39d95b35] FINISH, ConnectStoragePoolVDSCommand, log id: 35aff7e0 >2012-09-02 17:55:31,964 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-34) [39d95b35] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 17:55:31,964 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-34) [39d95b35] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:55:32,054 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-34) [2808669e] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 17:55:32,135 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-34) [2808669e] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5f7ff291 >2012-09-02 17:55:32,135 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-34) [2808669e] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 17:55:32,135 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-34) [2808669e] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5f7ff291 >2012-09-02 17:55:35,683 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-7) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=green-vdsa.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 17:55:42,179 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-91) [1129d67] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 17:55:43,003 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-91) [1129d67] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 13c0fde5 >2012-09-02 17:55:43,880 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-91) [1129d67] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 17:55:43,880 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-91) [1129d67] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 17:55:43,880 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-91) [1129d67] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:55:43,880 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-91) [1129d67] FINISH, ConnectStoragePoolVDSCommand, log id: 13c0fde5 >2012-09-02 17:55:43,880 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-91) [1129d67] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 17:55:43,880 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-91) [1129d67] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:55:43,953 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-91) [212eaddb] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 17:55:43,993 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-91) [212eaddb] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2bdcdb3a >2012-09-02 17:55:43,993 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-91) [212eaddb] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 17:55:43,993 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-91) [212eaddb] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2bdcdb3a >2012-09-02 17:55:45,733 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-11) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=green-vdsa.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 17:55:54,030 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [7d8f5d69] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 17:55:54,050 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-61) [7d8f5d69] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 6e359bb8 >2012-09-02 17:55:54,908 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-61) [7d8f5d69] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 17:55:54,908 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-61) [7d8f5d69] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 17:55:54,908 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-61) [7d8f5d69] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:55:54,909 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-61) [7d8f5d69] FINISH, ConnectStoragePoolVDSCommand, log id: 6e359bb8 >2012-09-02 17:55:54,909 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [7d8f5d69] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 17:55:54,909 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [7d8f5d69] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:55:55,004 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-61) [57348142] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 17:55:55,038 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-61) [57348142] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2f961e7c >2012-09-02 17:55:55,038 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [57348142] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 17:55:55,038 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-61) [57348142] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2f961e7c >2012-09-02 17:55:55,767 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-10) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=green-vdsa.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 17:56:05,266 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-6) [1dd5be77] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 17:56:05,503 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-6) [1dd5be77] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 5b49db0f >2012-09-02 17:56:05,803 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-9) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=green-vdsa.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 17:56:06,364 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-6) [1dd5be77] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 17:56:06,364 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-6) [1dd5be77] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 17:56:06,364 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-6) [1dd5be77] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:56:06,364 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-6) [1dd5be77] FINISH, ConnectStoragePoolVDSCommand, log id: 5b49db0f >2012-09-02 17:56:06,364 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-6) [1dd5be77] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 17:56:06,364 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-6) [1dd5be77] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:56:06,563 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-6) [23e79838] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 17:56:06,623 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-6) [23e79838] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3d4554d >2012-09-02 17:56:06,623 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-6) [23e79838] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 17:56:06,623 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-6) [23e79838] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3d4554d >2012-09-02 17:56:15,847 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-8) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=green-vdsa.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 17:56:16,681 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [47f4fd8c] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 17:56:16,703 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-14) [47f4fd8c] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 76156f52 >2012-09-02 17:56:17,559 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-14) [47f4fd8c] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 17:56:17,559 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-14) [47f4fd8c] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 17:56:17,559 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-14) [47f4fd8c] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:56:17,559 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-14) [47f4fd8c] FINISH, ConnectStoragePoolVDSCommand, log id: 76156f52 >2012-09-02 17:56:17,559 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [47f4fd8c] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 17:56:17,559 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [47f4fd8c] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:56:17,638 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-14) [37170097] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 17:56:17,673 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-14) [37170097] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3c13428d >2012-09-02 17:56:17,673 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [37170097] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 17:56:17,673 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-14) [37170097] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3c13428d >2012-09-02 17:56:25,883 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-6) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=green-vdsa.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 17:56:27,707 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [5a2190ec] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 17:56:27,956 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-30) [5a2190ec] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: f9f3581 >2012-09-02 17:56:28,823 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-30) [5a2190ec] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 17:56:28,824 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-30) [5a2190ec] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 17:56:28,824 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-30) [5a2190ec] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:56:28,824 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-30) [5a2190ec] FINISH, ConnectStoragePoolVDSCommand, log id: f9f3581 >2012-09-02 17:56:28,824 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [5a2190ec] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 17:56:28,824 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [5a2190ec] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:56:29,446 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-30) [51fed722] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 17:56:29,481 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-30) [51fed722] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7d20bd19 >2012-09-02 17:56:29,481 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [51fed722] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 17:56:29,481 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-30) [51fed722] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7d20bd19 >2012-09-02 17:56:35,918 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-3) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=green-vdsa.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 17:56:39,547 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [6dc69ea2] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 17:56:39,571 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-50) [6dc69ea2] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 629e067a >2012-09-02 17:56:40,446 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-50) [6dc69ea2] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 17:56:40,447 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-50) [6dc69ea2] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 17:56:40,447 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-50) [6dc69ea2] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:56:40,447 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-50) [6dc69ea2] FINISH, ConnectStoragePoolVDSCommand, log id: 629e067a >2012-09-02 17:56:40,447 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [6dc69ea2] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 17:56:40,447 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [6dc69ea2] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:56:42,277 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-50) [5398543e] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 17:56:42,430 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-50) [5398543e] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 166a18db >2012-09-02 17:56:42,430 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [5398543e] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 17:56:42,430 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-50) [5398543e] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 166a18db >2012-09-02 17:56:45,958 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-7) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=green-vdsa.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 17:56:52,462 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [ba9ffb2] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 17:56:52,486 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-71) [ba9ffb2] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 710f2500 >2012-09-02 17:56:53,359 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-71) [ba9ffb2] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 17:56:53,359 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-71) [ba9ffb2] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 17:56:53,359 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-71) [ba9ffb2] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:56:53,359 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-71) [ba9ffb2] FINISH, ConnectStoragePoolVDSCommand, log id: 710f2500 >2012-09-02 17:56:53,359 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [ba9ffb2] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 17:56:53,359 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [ba9ffb2] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:56:53,432 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-71) [5bcf149a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 17:56:53,470 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-71) [5bcf149a] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6eee3907 >2012-09-02 17:56:53,470 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [5bcf149a] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 17:56:53,470 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-71) [5bcf149a] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6eee3907 >2012-09-02 17:56:55,995 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-11) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=green-vdsa.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 17:57:03,507 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-100) [6dae079f] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 17:57:03,848 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-100) [6dae079f] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 3595203a >2012-09-02 17:57:04,718 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-100) [6dae079f] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 17:57:04,719 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-100) [6dae079f] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 17:57:04,719 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-100) [6dae079f] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:57:04,719 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-100) [6dae079f] FINISH, ConnectStoragePoolVDSCommand, log id: 3595203a >2012-09-02 17:57:04,719 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-100) [6dae079f] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 17:57:04,719 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-100) [6dae079f] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:57:04,796 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-100) [1f190ccd] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 17:57:04,832 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-100) [1f190ccd] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6ed76066 >2012-09-02 17:57:04,832 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-100) [1f190ccd] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 17:57:04,832 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-100) [1f190ccd] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6ed76066 >2012-09-02 17:57:06,030 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-10) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=green-vdsa.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 17:57:14,864 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [52fbf215] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 17:57:14,895 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-8) [52fbf215] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: c64cbc >2012-09-02 17:57:15,758 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-8) [52fbf215] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 17:57:15,758 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-8) [52fbf215] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 17:57:15,758 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-8) [52fbf215] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:57:15,758 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-8) [52fbf215] FINISH, ConnectStoragePoolVDSCommand, log id: c64cbc >2012-09-02 17:57:15,758 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [52fbf215] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 17:57:15,758 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [52fbf215] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:57:15,815 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-8) [18420dcd] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 17:57:15,851 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-8) [18420dcd] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1f93bc09 >2012-09-02 17:57:15,851 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [18420dcd] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 17:57:15,851 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-8) [18420dcd] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1f93bc09 >2012-09-02 17:57:16,066 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-1) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=green-vdsa.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 17:57:25,888 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-28) [164d9fbc] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 17:57:25,905 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-28) [164d9fbc] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 7d46742c >2012-09-02 17:57:26,130 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-8) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=green-vdsa.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 17:57:26,778 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-28) [164d9fbc] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 17:57:26,778 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-28) [164d9fbc] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 17:57:26,778 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-28) [164d9fbc] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:57:26,778 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-28) [164d9fbc] FINISH, ConnectStoragePoolVDSCommand, log id: 7d46742c >2012-09-02 17:57:26,778 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-28) [164d9fbc] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 17:57:26,778 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-28) [164d9fbc] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:57:27,267 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-28) [1d49a7b8] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 17:57:27,300 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-28) [1d49a7b8] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 316a0731 >2012-09-02 17:57:27,300 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-28) [1d49a7b8] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 17:57:27,300 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-28) [1d49a7b8] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 316a0731 >2012-09-02 17:57:36,165 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-6) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=green-vdsa.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 17:57:37,339 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [51fed722] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 17:57:37,371 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-30) [51fed722] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 7783bca3 >2012-09-02 17:57:38,248 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-30) [51fed722] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 17:57:38,248 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-30) [51fed722] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 17:57:38,248 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-30) [51fed722] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:57:38,248 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-30) [51fed722] FINISH, ConnectStoragePoolVDSCommand, log id: 7783bca3 >2012-09-02 17:57:38,248 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [51fed722] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 17:57:38,248 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [51fed722] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:57:38,355 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-30) [7097f0e4] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 17:57:38,399 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-30) [7097f0e4] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2eb3901c >2012-09-02 17:57:38,399 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [7097f0e4] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 17:57:38,399 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-30) [7097f0e4] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2eb3901c >2012-09-02 17:57:46,201 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-3) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=green-vdsa.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 17:57:48,474 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-58) [7d786bb3] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 17:57:48,496 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-58) [7d786bb3] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 5c468a02 >2012-09-02 17:57:49,351 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-58) [7d786bb3] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 17:57:49,351 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-58) [7d786bb3] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 17:57:49,351 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-58) [7d786bb3] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:57:49,351 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-58) [7d786bb3] FINISH, ConnectStoragePoolVDSCommand, log id: 5c468a02 >2012-09-02 17:57:49,351 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-58) [7d786bb3] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 17:57:49,351 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-58) [7d786bb3] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:57:49,425 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-58) [8e75c63] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 17:57:49,482 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-58) [8e75c63] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7b2a22ca >2012-09-02 17:57:49,482 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-58) [8e75c63] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 17:57:49,483 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-58) [8e75c63] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7b2a22ca >2012-09-02 17:57:56,236 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-7) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=green-vdsa.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 17:57:59,520 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [17f32511] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 17:57:59,552 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-84) [17f32511] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 62c2ccc7 >2012-09-02 17:58:00,584 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-84) [17f32511] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 17:58:00,585 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-84) [17f32511] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 17:58:00,585 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-84) [17f32511] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:58:00,585 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-84) [17f32511] FINISH, ConnectStoragePoolVDSCommand, log id: 62c2ccc7 >2012-09-02 17:58:00,585 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [17f32511] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 17:58:00,585 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [17f32511] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:58:00,711 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-84) [7f2f274] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 17:58:00,748 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-84) [7f2f274] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2798731e >2012-09-02 17:58:00,748 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [7f2f274] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 17:58:00,748 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-84) [7f2f274] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2798731e >2012-09-02 17:58:06,272 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-11) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=green-vdsa.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 17:58:10,790 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-3) hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 17:58:11,295 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-3) START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 73b43032 >2012-09-02 17:58:12,164 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-3) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 17:58:12,164 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-3) Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 17:58:12,164 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-3) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:58:12,164 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-3) FINISH, ConnectStoragePoolVDSCommand, log id: 73b43032 >2012-09-02 17:58:12,164 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-3) IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 17:58:12,165 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-3) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:58:12,583 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-3) [3eebd88c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 17:58:12,621 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-3) [3eebd88c] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 40a9ae56 >2012-09-02 17:58:12,621 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-3) [3eebd88c] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 17:58:12,621 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-3) [3eebd88c] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 40a9ae56 >2012-09-02 17:58:16,307 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-5) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=green-vdsa.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 17:58:22,663 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) [6709ab6a] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 17:58:22,685 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-13) [6709ab6a] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 5ec8ce76 >2012-09-02 17:58:23,545 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-13) [6709ab6a] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 17:58:23,545 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-13) [6709ab6a] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 17:58:23,545 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-13) [6709ab6a] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:58:23,545 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-13) [6709ab6a] FINISH, ConnectStoragePoolVDSCommand, log id: 5ec8ce76 >2012-09-02 17:58:23,545 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) [6709ab6a] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 17:58:23,545 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) [6709ab6a] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:58:23,635 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-13) [188d4eb1] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 17:58:23,670 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-13) [188d4eb1] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: ce003f6 >2012-09-02 17:58:23,670 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) [188d4eb1] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 17:58:23,671 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-13) [188d4eb1] FINISH, MarkPoolInReconstructModeVDSCommand, log id: ce003f6 >2012-09-02 17:58:26,333 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-1) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=green-vdsa.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 17:58:33,703 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [7648219e] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 17:58:33,720 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-23) [7648219e] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 4a4e1f52 >2012-09-02 17:58:34,589 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-23) [7648219e] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 17:58:34,589 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-23) [7648219e] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 17:58:34,589 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-23) [7648219e] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:58:34,589 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-23) [7648219e] FINISH, ConnectStoragePoolVDSCommand, log id: 4a4e1f52 >2012-09-02 17:58:34,589 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [7648219e] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 17:58:34,589 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [7648219e] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:58:35,637 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-23) [1362cca1] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 17:58:35,721 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-23) [1362cca1] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3b719132 >2012-09-02 17:58:35,721 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [1362cca1] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 17:58:35,721 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-23) [1362cca1] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3b719132 >2012-09-02 17:58:36,368 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-8) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=green-vdsa.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 17:58:45,761 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) [dbe32c9] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 17:58:45,779 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-60) [dbe32c9] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 7d37ef46 >2012-09-02 17:58:46,403 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-6) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=green-vdsa.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 17:58:46,615 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-60) [dbe32c9] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 17:58:46,615 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-60) [dbe32c9] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 17:58:46,615 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-60) [dbe32c9] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:58:46,615 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-60) [dbe32c9] FINISH, ConnectStoragePoolVDSCommand, log id: 7d37ef46 >2012-09-02 17:58:46,615 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) [dbe32c9] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 17:58:46,615 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) [dbe32c9] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:58:46,759 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-60) [24ffebc0] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 17:58:46,788 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-60) [24ffebc0] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 13301756 >2012-09-02 17:58:46,788 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) [24ffebc0] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 17:58:46,788 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-60) [24ffebc0] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 13301756 >2012-09-02 17:58:56,443 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-3) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=green-vdsa.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 17:58:56,822 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-58) [8e75c63] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 17:58:56,838 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-58) [8e75c63] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 3f641a8e >2012-09-02 17:58:57,708 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-58) [8e75c63] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 17:58:57,708 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-58) [8e75c63] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 17:58:57,708 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-58) [8e75c63] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:58:57,708 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-58) [8e75c63] FINISH, ConnectStoragePoolVDSCommand, log id: 3f641a8e >2012-09-02 17:58:57,708 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-58) [8e75c63] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 17:58:57,708 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-58) [8e75c63] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:58:57,787 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-58) [61be7fc9] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 17:58:57,820 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-58) [61be7fc9] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 54f2c0a6 >2012-09-02 17:58:57,820 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-58) [61be7fc9] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 17:58:57,820 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-58) [61be7fc9] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 54f2c0a6 >2012-09-02 17:59:06,486 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-7) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=green-vdsa.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 17:59:07,877 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [7f2f274] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 17:59:07,895 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-84) [7f2f274] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 24757167 >2012-09-02 17:59:08,762 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-84) [7f2f274] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 17:59:08,762 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-84) [7f2f274] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 17:59:08,762 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-84) [7f2f274] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:59:08,762 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-84) [7f2f274] FINISH, ConnectStoragePoolVDSCommand, log id: 24757167 >2012-09-02 17:59:08,762 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [7f2f274] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 17:59:08,762 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [7f2f274] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:59:08,895 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-84) [1571730f] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 17:59:08,961 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-84) [1571730f] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 74e0f41a >2012-09-02 17:59:08,961 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [1571730f] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 17:59:08,961 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-84) [1571730f] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 74e0f41a >2012-09-02 17:59:16,621 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-4) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=green-vdsa.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 17:59:19,009 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-93) [361e27ee] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 17:59:19,040 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-93) [361e27ee] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 6f2ca585 >2012-09-02 17:59:19,911 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-93) [361e27ee] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 17:59:19,911 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-93) [361e27ee] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 17:59:19,911 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-93) [361e27ee] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:59:19,911 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-93) [361e27ee] FINISH, ConnectStoragePoolVDSCommand, log id: 6f2ca585 >2012-09-02 17:59:19,911 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-93) [361e27ee] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 17:59:19,911 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-93) [361e27ee] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:59:20,044 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-93) [6721d3d3] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 17:59:20,077 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-93) [6721d3d3] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 19b8c495 >2012-09-02 17:59:20,077 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-93) [6721d3d3] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 17:59:20,077 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-93) [6721d3d3] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 19b8c495 >2012-09-02 17:59:26,657 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-5) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=green-vdsa.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 17:59:30,221 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-64) [5bdf963c] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 17:59:30,646 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-64) [5bdf963c] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 397fb98b >2012-09-02 17:59:31,504 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-64) [5bdf963c] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 17:59:31,504 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-64) [5bdf963c] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 17:59:31,504 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-64) [5bdf963c] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:59:31,504 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-64) [5bdf963c] FINISH, ConnectStoragePoolVDSCommand, log id: 397fb98b >2012-09-02 17:59:31,504 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-64) [5bdf963c] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 17:59:31,504 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-64) [5bdf963c] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:59:31,610 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-64) [411b2322] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 17:59:31,663 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-64) [411b2322] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: b8208d4 >2012-09-02 17:59:31,663 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-64) [411b2322] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 17:59:31,663 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-64) [411b2322] FINISH, MarkPoolInReconstructModeVDSCommand, log id: b8208d4 >2012-09-02 17:59:36,692 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-1) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=green-vdsa.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 17:59:41,719 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) [227b1500] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 17:59:41,743 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-31) [227b1500] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 500e9091 >2012-09-02 17:59:42,604 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-31) [227b1500] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 17:59:42,604 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-31) [227b1500] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 17:59:42,604 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-31) [227b1500] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:59:42,604 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-31) [227b1500] FINISH, ConnectStoragePoolVDSCommand, log id: 500e9091 >2012-09-02 17:59:42,604 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) [227b1500] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 17:59:42,604 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) [227b1500] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:59:42,711 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-31) [4da23022] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 17:59:42,757 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-31) [4da23022] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7f3f0f4b >2012-09-02 17:59:42,757 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) [4da23022] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 17:59:42,757 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-31) [4da23022] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7f3f0f4b >2012-09-02 17:59:46,727 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-8) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=green-vdsa.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 17:59:52,813 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [1cf6d845] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 17:59:52,830 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-42) [1cf6d845] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 101ac4e4 >2012-09-02 17:59:53,678 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-42) [1cf6d845] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 17:59:53,678 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-42) [1cf6d845] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 17:59:53,678 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-42) [1cf6d845] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:59:53,678 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-42) [1cf6d845] FINISH, ConnectStoragePoolVDSCommand, log id: 101ac4e4 >2012-09-02 17:59:53,679 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [1cf6d845] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 17:59:53,679 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [1cf6d845] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 17:59:53,761 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-42) [703e4e76] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 17:59:53,797 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-42) [703e4e76] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 251ab4b5 >2012-09-02 17:59:53,798 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [703e4e76] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 17:59:53,798 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-42) [703e4e76] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 251ab4b5 >2012-09-02 17:59:56,762 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-6) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=green-vdsa.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:00:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-48) [1ea6b2c6] Autorecovering hosts is disabled, skipping >2012-09-02 18:00:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-48) [1ea6b2c6] Autorecovering storage domains is disabled, skipping >2012-09-02 18:00:03,842 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-68) hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 18:00:03,913 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-68) START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 6e29c333 >2012-09-02 18:00:05,035 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-68) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 18:00:05,035 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-68) Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 18:00:05,035 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-68) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 18:00:05,035 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-68) FINISH, ConnectStoragePoolVDSCommand, log id: 6e29c333 >2012-09-02 18:00:05,035 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-68) IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:00:05,035 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-68) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 18:00:05,348 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-68) [55173ac5] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:00:05,418 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-68) [55173ac5] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: f851273 >2012-09-02 18:00:05,418 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-68) [55173ac5] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:00:05,418 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-68) [55173ac5] FINISH, MarkPoolInReconstructModeVDSCommand, log id: f851273 >2012-09-02 18:00:06,798 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-3) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=green-vdsa.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:00:15,476 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-91) [212eaddb] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 18:00:15,525 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-91) [212eaddb] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 2659d759 >2012-09-02 18:00:16,515 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-91) [212eaddb] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 18:00:16,515 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-91) [212eaddb] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 18:00:16,516 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-91) [212eaddb] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 18:00:16,516 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-91) [212eaddb] FINISH, ConnectStoragePoolVDSCommand, log id: 2659d759 >2012-09-02 18:00:16,516 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-91) [212eaddb] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:00:16,516 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-91) [212eaddb] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 18:00:16,622 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-91) [52fa7b4e] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:00:16,661 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-91) [52fa7b4e] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 374a9a29 >2012-09-02 18:00:16,661 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-91) [52fa7b4e] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:00:16,661 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-91) [52fa7b4e] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 374a9a29 >2012-09-02 18:00:17,511 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-2) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=green-vdsa.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:00:26,760 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-88) hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 18:00:26,778 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-88) START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 74a46388 >2012-09-02 18:00:27,546 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-4) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=green-vdsa.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:00:27,653 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-88) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 18:00:27,653 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-88) Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 18:00:27,653 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-88) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 18:00:27,653 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-88) FINISH, ConnectStoragePoolVDSCommand, log id: 74a46388 >2012-09-02 18:00:27,653 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-88) IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:00:27,653 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-88) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 18:00:28,272 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-88) [323af124] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:00:28,309 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-88) [323af124] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7339cf4b >2012-09-02 18:00:28,309 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-88) [323af124] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:00:28,309 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-88) [323af124] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7339cf4b >2012-09-02 18:00:37,585 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-5) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=green-vdsa.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:00:38,378 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [1f04073e] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 18:00:38,400 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-1) [1f04073e] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 55630346 >2012-09-02 18:00:39,260 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-1) [1f04073e] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 18:00:39,260 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-1) [1f04073e] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 18:00:39,260 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-1) [1f04073e] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 18:00:39,260 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-1) [1f04073e] FINISH, ConnectStoragePoolVDSCommand, log id: 55630346 >2012-09-02 18:00:39,260 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [1f04073e] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:00:39,260 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [1f04073e] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 18:00:40,649 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-1) [2fbdf05f] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:00:42,014 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-1) [2fbdf05f] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 112f5cd9 >2012-09-02 18:00:42,014 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [2fbdf05f] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:00:42,014 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-1) [2fbdf05f] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 112f5cd9 >2012-09-02 18:00:47,618 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-1) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=green-vdsa.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:00:52,070 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) [4da23022] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 18:00:52,082 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-31) [4da23022] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 5222cc3a >2012-09-02 18:00:52,971 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-31) [4da23022] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 18:00:52,971 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-31) [4da23022] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 18:00:52,971 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-31) [4da23022] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 18:00:52,971 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-31) [4da23022] FINISH, ConnectStoragePoolVDSCommand, log id: 5222cc3a >2012-09-02 18:00:52,971 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) [4da23022] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:00:52,971 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) [4da23022] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 18:00:53,086 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-31) [3c79dd4f] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:00:53,137 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-31) [3c79dd4f] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 71038dd0 >2012-09-02 18:00:53,137 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) [3c79dd4f] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:00:53,137 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-31) [3c79dd4f] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 71038dd0 >2012-09-02 18:00:57,658 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-8) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=green-vdsa.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:01:03,181 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [6c3830ce] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 18:01:03,250 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-39) [6c3830ce] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 45f0b994 >2012-09-02 18:01:04,205 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-39) [6c3830ce] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 18:01:04,205 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-39) [6c3830ce] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 18:01:04,205 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-39) [6c3830ce] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 18:01:04,205 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-39) [6c3830ce] FINISH, ConnectStoragePoolVDSCommand, log id: 45f0b994 >2012-09-02 18:01:04,205 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [6c3830ce] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:01:04,205 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [6c3830ce] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 18:01:04,717 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-39) [18c0d8a4] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:01:04,835 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-39) [18c0d8a4] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3b7974b9 >2012-09-02 18:01:04,835 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [18c0d8a4] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:01:04,835 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-39) [18c0d8a4] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3b7974b9 >2012-09-02 18:01:07,821 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-6) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=green-vdsa.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:01:14,873 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-62) [3ea43f70] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 18:01:14,969 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-62) [3ea43f70] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 1c76749b >2012-09-02 18:01:15,842 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-62) [3ea43f70] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 18:01:15,842 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-62) [3ea43f70] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 18:01:15,842 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-62) [3ea43f70] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 18:01:15,842 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-62) [3ea43f70] FINISH, ConnectStoragePoolVDSCommand, log id: 1c76749b >2012-09-02 18:01:15,842 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-62) [3ea43f70] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:01:15,842 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-62) [3ea43f70] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 18:01:15,948 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-62) [320e0002] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:01:15,983 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-62) [320e0002] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5432f370 >2012-09-02 18:01:15,983 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-62) [320e0002] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:01:15,983 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-62) [320e0002] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5432f370 >2012-09-02 18:01:17,856 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-9) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=green-vdsa.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:01:26,022 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-76) [a637d5a] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 18:01:26,040 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-76) [a637d5a] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 54915e5c >2012-09-02 18:01:26,931 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-76) [a637d5a] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 18:01:26,931 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-76) [a637d5a] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 18:01:26,931 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-76) [a637d5a] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 18:01:26,931 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-76) [a637d5a] FINISH, ConnectStoragePoolVDSCommand, log id: 54915e5c >2012-09-02 18:01:26,931 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-76) [a637d5a] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:01:26,931 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-76) [a637d5a] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 18:01:27,525 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-76) [3f33ec20] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:01:27,572 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-76) [3f33ec20] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 25d24fa9 >2012-09-02 18:01:27,572 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-76) [3f33ec20] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:01:27,572 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-76) [3f33ec20] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 25d24fa9 >2012-09-02 18:01:27,920 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-2) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=green-vdsa.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:01:37,610 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-95) [4952ea3d] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 18:01:37,833 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-95) [4952ea3d] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 510345 >2012-09-02 18:01:37,951 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-4) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=green-vdsa.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:01:38,709 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-95) [4952ea3d] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 18:01:38,709 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-95) [4952ea3d] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 18:01:38,709 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-95) [4952ea3d] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 18:01:38,709 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-95) [4952ea3d] FINISH, ConnectStoragePoolVDSCommand, log id: 510345 >2012-09-02 18:01:38,710 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-95) [4952ea3d] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:01:38,710 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-95) [4952ea3d] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 18:01:38,987 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-95) [20dcf559] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:01:39,061 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-95) [20dcf559] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 524745cf >2012-09-02 18:01:39,061 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-95) [20dcf559] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:01:39,061 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-95) [20dcf559] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 524745cf >2012-09-02 18:01:47,991 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-5) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=green-vdsa.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:01:49,130 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-20) [330b8136] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 18:01:49,161 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-20) [330b8136] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 5cca7166 >2012-09-02 18:01:50,035 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-20) [330b8136] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 18:01:50,035 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-20) [330b8136] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 18:01:50,035 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-20) [330b8136] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 18:01:50,035 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-20) [330b8136] FINISH, ConnectStoragePoolVDSCommand, log id: 5cca7166 >2012-09-02 18:01:50,035 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-20) [330b8136] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:01:50,035 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-20) [330b8136] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 18:01:50,551 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-20) [6f64cdb5] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:01:50,591 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-20) [6f64cdb5] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 11be79f8 >2012-09-02 18:01:50,591 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-20) [6f64cdb5] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:01:50,591 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-20) [6f64cdb5] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 11be79f8 >2012-09-02 18:01:58,027 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-1) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=green-vdsa.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:02:00,629 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [a89e0e7] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 18:02:00,645 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-27) [a89e0e7] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 2f21d671 >2012-09-02 18:02:01,501 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-27) [a89e0e7] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 18:02:01,501 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-27) [a89e0e7] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 18:02:01,502 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-27) [a89e0e7] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 18:02:01,502 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-27) [a89e0e7] FINISH, ConnectStoragePoolVDSCommand, log id: 2f21d671 >2012-09-02 18:02:01,502 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [a89e0e7] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:02:01,502 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [a89e0e7] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 18:02:01,579 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-27) [61ec3f50] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:02:01,612 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-27) [61ec3f50] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 70df8042 >2012-09-02 18:02:01,613 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [61ec3f50] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:02:01,613 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-27) [61ec3f50] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 70df8042 >2012-09-02 18:02:08,062 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-8) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=green-vdsa.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:02:11,698 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-43) [35748f38] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 18:02:11,714 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-43) [35748f38] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 58f7e751 >2012-09-02 18:02:12,590 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-43) [35748f38] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 18:02:12,590 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-43) [35748f38] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 18:02:12,590 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-43) [35748f38] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 18:02:12,590 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-43) [35748f38] FINISH, ConnectStoragePoolVDSCommand, log id: 58f7e751 >2012-09-02 18:02:12,590 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-43) [35748f38] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:02:12,590 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-43) [35748f38] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 18:02:13,818 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-43) [3ecfc237] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:02:13,860 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-43) [3ecfc237] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 35e46388 >2012-09-02 18:02:13,860 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-43) [3ecfc237] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:02:13,860 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-43) [3ecfc237] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 35e46388 >2012-09-02 18:02:18,097 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-11) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=green-vdsa.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:02:23,905 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-34) [2808669e] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 18:02:23,927 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-34) [2808669e] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 6f2a00f5 >2012-09-02 18:02:24,784 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-34) [2808669e] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 18:02:24,784 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-34) [2808669e] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 18:02:24,784 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-34) [2808669e] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 18:02:24,784 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-34) [2808669e] FINISH, ConnectStoragePoolVDSCommand, log id: 6f2a00f5 >2012-09-02 18:02:24,784 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-34) [2808669e] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:02:24,784 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-34) [2808669e] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 18:02:24,883 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-34) [58a0355c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:02:24,922 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-34) [58a0355c] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 879cae9 >2012-09-02 18:02:24,923 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-34) [58a0355c] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:02:24,923 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-34) [58a0355c] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 879cae9 >2012-09-02 18:02:28,133 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-9) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=green-vdsa.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:02:34,975 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-75) [1b3d0324] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 18:02:35,007 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-75) [1b3d0324] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 74c8c5d4 >2012-09-02 18:02:35,877 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-75) [1b3d0324] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 18:02:35,877 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-75) [1b3d0324] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 18:02:35,878 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-75) [1b3d0324] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 18:02:35,878 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-75) [1b3d0324] FINISH, ConnectStoragePoolVDSCommand, log id: 74c8c5d4 >2012-09-02 18:02:35,878 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-75) [1b3d0324] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:02:35,878 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-75) [1b3d0324] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 18:02:35,989 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-75) [602f07dc] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:02:36,073 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-75) [602f07dc] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 15a17c41 >2012-09-02 18:02:36,073 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-75) [602f07dc] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:02:36,073 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-75) [602f07dc] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 15a17c41 >2012-09-02 18:02:38,168 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-2) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=green-vdsa.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:02:46,114 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [57348142] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 18:02:46,131 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-61) [57348142] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 4c87296f >2012-09-02 18:02:47,008 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-61) [57348142] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 18:02:47,008 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-61) [57348142] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 18:02:47,008 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-61) [57348142] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 18:02:47,008 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-61) [57348142] FINISH, ConnectStoragePoolVDSCommand, log id: 4c87296f >2012-09-02 18:02:47,008 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [57348142] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:02:47,008 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [57348142] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 18:02:47,080 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-61) [77cb3047] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:02:47,124 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-61) [77cb3047] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5b4394ba >2012-09-02 18:02:47,124 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [77cb3047] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:02:47,124 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-61) [77cb3047] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5b4394ba >2012-09-02 18:02:48,200 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-4) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=green-vdsa.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:02:57,158 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [40eb8ded] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 18:02:57,178 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-5) [40eb8ded] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: bdf9644 >2012-09-02 18:02:58,050 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-5) [40eb8ded] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 18:02:58,050 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-5) [40eb8ded] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 18:02:58,050 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-5) [40eb8ded] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 18:02:58,050 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-5) [40eb8ded] FINISH, ConnectStoragePoolVDSCommand, log id: bdf9644 >2012-09-02 18:02:58,050 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [40eb8ded] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:02:58,050 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [40eb8ded] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 18:02:58,126 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-5) [17d20690] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:02:58,164 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-5) [17d20690] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6b73eb12 >2012-09-02 18:02:58,164 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [17d20690] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:02:58,164 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-5) [17d20690] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6b73eb12 >2012-09-02 18:02:58,228 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-5) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=green-vdsa.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:03:08,237 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [22836716] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 18:03:08,264 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-1) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=green-vdsa.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:03:08,489 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-16) [22836716] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 615c6b5e >2012-09-02 18:03:09,370 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-16) [22836716] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 18:03:09,370 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-16) [22836716] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 18:03:09,370 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-16) [22836716] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 18:03:09,370 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-16) [22836716] FINISH, ConnectStoragePoolVDSCommand, log id: 615c6b5e >2012-09-02 18:03:09,370 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [22836716] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:03:09,370 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [22836716] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 18:03:09,479 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-16) [1ca25f7e] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:03:09,527 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-16) [1ca25f7e] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 36010400 >2012-09-02 18:03:09,527 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [1ca25f7e] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:03:09,527 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-16) [1ca25f7e] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 36010400 >2012-09-02 18:03:18,301 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-3) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=green-vdsa.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:03:19,564 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) [2a140678] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 18:03:19,586 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-41) [2a140678] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 5745fa74 >2012-09-02 18:03:20,473 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-41) [2a140678] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 18:03:20,473 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-41) [2a140678] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 18:03:20,473 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-41) [2a140678] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 18:03:20,473 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-41) [2a140678] FINISH, ConnectStoragePoolVDSCommand, log id: 5745fa74 >2012-09-02 18:03:20,473 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) [2a140678] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:03:20,473 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) [2a140678] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 18:03:20,562 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-41) [727ad312] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:03:20,601 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-41) [727ad312] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 31a1f472 >2012-09-02 18:03:20,601 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) [727ad312] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:03:20,601 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-41) [727ad312] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 31a1f472 >2012-09-02 18:03:28,338 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-11) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=green-vdsa.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:03:30,650 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-45) [4a22f461] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 18:03:30,666 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-45) [4a22f461] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 17efcd6d >2012-09-02 18:03:31,536 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-45) [4a22f461] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 18:03:31,536 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-45) [4a22f461] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 18:03:31,536 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-45) [4a22f461] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 18:03:31,536 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-45) [4a22f461] FINISH, ConnectStoragePoolVDSCommand, log id: 17efcd6d >2012-09-02 18:03:31,536 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-45) [4a22f461] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:03:31,536 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-45) [4a22f461] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 18:03:31,612 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-45) [77216586] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:03:31,656 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-45) [77216586] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6f8d47b3 >2012-09-02 18:03:31,656 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-45) [77216586] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:03:31,656 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-45) [77216586] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6f8d47b3 >2012-09-02 18:03:38,373 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-9) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=green-vdsa.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:03:41,689 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) [5dc6965d] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 18:03:41,707 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-65) [5dc6965d] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 68d518bb >2012-09-02 18:03:42,033 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-69) [39fd24f3] DbUserCacheManager::refreshAllUserData() - entered >2012-09-02 18:03:42,582 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-65) [5dc6965d] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 18:03:42,582 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-65) [5dc6965d] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 18:03:42,582 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-65) [5dc6965d] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 18:03:42,582 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-65) [5dc6965d] FINISH, ConnectStoragePoolVDSCommand, log id: 68d518bb >2012-09-02 18:03:42,583 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) [5dc6965d] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:03:42,583 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) [5dc6965d] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 18:03:43,648 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-65) [51b0a65e] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:03:43,705 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-65) [51b0a65e] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 76053796 >2012-09-02 18:03:43,705 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) [51b0a65e] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:03:43,705 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-65) [51b0a65e] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 76053796 >2012-09-02 18:03:48,409 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-2) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=green-vdsa.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:03:53,749 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [5aabc0e5] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 18:03:53,803 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-77) [5aabc0e5] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 3ab3533b >2012-09-02 18:03:54,678 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-77) [5aabc0e5] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 18:03:54,678 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-77) [5aabc0e5] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 18:03:54,678 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-77) [5aabc0e5] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 18:03:54,678 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-77) [5aabc0e5] FINISH, ConnectStoragePoolVDSCommand, log id: 3ab3533b >2012-09-02 18:03:54,678 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [5aabc0e5] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:03:54,678 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [5aabc0e5] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 18:03:54,806 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-77) [6ae6e3db] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:03:54,845 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-77) [6ae6e3db] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7c55c4cb >2012-09-02 18:03:54,846 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [6ae6e3db] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:03:54,846 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-77) [6ae6e3db] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7c55c4cb >2012-09-02 18:03:56,783 INFO [org.ovirt.engine.core.bll.LoginAdminUserCommand] (ajp-/127.0.0.1:8009-5) Checking if user admin@internal is an admin, result true >2012-09-02 18:03:56,783 INFO [org.ovirt.engine.core.bll.LoginAdminUserCommand] (ajp-/127.0.0.1:8009-5) Running command: LoginAdminUserCommand internal: false. >2012-09-02 18:03:57,575 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8009-1) calling GetConfigurationValueQuery (CustomPublicConfig_AppsWebSite) with null version, using default general for version >2012-09-02 18:03:58,431 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-6) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=green-vdsa.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:04:04,896 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-4) [2eec22b4] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 18:04:04,985 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-4) [2eec22b4] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 4240244e >2012-09-02 18:04:05,867 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-4) [2eec22b4] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 18:04:05,867 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-4) [2eec22b4] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 18:04:05,867 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-4) [2eec22b4] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 18:04:05,867 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-4) [2eec22b4] FINISH, ConnectStoragePoolVDSCommand, log id: 4240244e >2012-09-02 18:04:05,867 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-4) [2eec22b4] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:04:05,867 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-4) [2eec22b4] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 18:04:06,439 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-4) [79d32fb4] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:04:06,488 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-4) [79d32fb4] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 276d177b >2012-09-02 18:04:06,488 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-4) [79d32fb4] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:04:06,488 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-4) [79d32fb4] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 276d177b >2012-09-02 18:04:08,467 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-4) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=green-vdsa.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:04:16,550 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-19) [229837ce] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 18:04:16,565 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-19) [229837ce] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: fe82bf9 >2012-09-02 18:04:17,436 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-19) [229837ce] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 18:04:17,436 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-19) [229837ce] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 18:04:17,436 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-19) [229837ce] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 18:04:17,436 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-19) [229837ce] FINISH, ConnectStoragePoolVDSCommand, log id: fe82bf9 >2012-09-02 18:04:17,436 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-19) [229837ce] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:04:17,436 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-19) [229837ce] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 18:04:17,557 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-19) [66e5caf9] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:04:17,605 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-19) [66e5caf9] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 617969f6 >2012-09-02 18:04:17,605 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-19) [66e5caf9] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:04:17,606 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-19) [66e5caf9] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 617969f6 >2012-09-02 18:04:27,652 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-24) [31e52f20] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 18:04:28,336 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-24) [31e52f20] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 3369dbd5 >2012-09-02 18:04:29,209 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-24) [31e52f20] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 18:04:29,209 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-24) [31e52f20] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 18:04:29,209 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-24) [31e52f20] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 18:04:29,209 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-24) [31e52f20] FINISH, ConnectStoragePoolVDSCommand, log id: 3369dbd5 >2012-09-02 18:04:29,209 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-24) [31e52f20] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:04:29,209 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-24) [31e52f20] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 18:04:30,335 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-24) [38229306] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:04:30,621 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-24) [38229306] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 24b11519 >2012-09-02 18:04:30,621 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-24) [38229306] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:04:30,621 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-24) [38229306] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 24b11519 >2012-09-02 18:04:41,550 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [1ea6b2c6] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 18:04:42,769 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-48) [1ea6b2c6] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 5bc77212 >2012-09-02 18:04:43,635 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-48) [1ea6b2c6] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 18:04:43,635 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-48) [1ea6b2c6] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 18:04:43,635 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-48) [1ea6b2c6] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 18:04:43,635 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-48) [1ea6b2c6] FINISH, ConnectStoragePoolVDSCommand, log id: 5bc77212 >2012-09-02 18:04:43,635 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [1ea6b2c6] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:04:43,635 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [1ea6b2c6] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 18:04:44,547 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-48) [cde8241] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:04:44,579 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-48) [cde8241] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 29846a25 >2012-09-02 18:04:44,579 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [cde8241] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:04:44,579 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-48) [cde8241] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 29846a25 >2012-09-02 18:04:54,615 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) [51b0a65e] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 18:04:54,632 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-65) [51b0a65e] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: b3a81d5 >2012-09-02 18:04:55,532 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-65) [51b0a65e] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 18:04:55,532 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-65) [51b0a65e] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 18:04:55,532 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-65) [51b0a65e] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 18:04:55,532 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-65) [51b0a65e] FINISH, ConnectStoragePoolVDSCommand, log id: b3a81d5 >2012-09-02 18:04:55,532 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) [51b0a65e] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:04:55,532 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) [51b0a65e] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 18:04:55,627 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-65) [742a9b47] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:04:55,661 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-65) [742a9b47] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 61372ec0 >2012-09-02 18:04:55,661 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) [742a9b47] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:04:55,661 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-65) [742a9b47] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 61372ec0 >2012-09-02 18:04:57,969 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (ajp-/127.0.0.1:8009-5) Lock Acquired to object EngineLock [exclusiveLocks= key: 3880c572-f4f0-11e1-9a52-001a4a169705 value: VDS >, sharedLocks= ] >2012-09-02 18:04:58,042 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (pool-4-thread-50) [3b487353] Running command: ActivateVdsCommand internal: false. Entities affected : ID: 3880c572-f4f0-11e1-9a52-001a4a169705 Type: VDS >2012-09-02 18:04:58,046 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-50) [3b487353] START, SetVdsStatusVDSCommand(vdsId = 3880c572-f4f0-11e1-9a52-001a4a169705, status=Unassigned, nonOperationalReason=NONE), log id: 346e51fa >2012-09-02 18:04:58,048 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-50) [3b487353] FINISH, SetVdsStatusVDSCommand, log id: 346e51fa >2012-09-02 18:04:58,067 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (pool-4-thread-50) [3b487353] START, ActivateVdsVDSCommand(vdsId = 3880c572-f4f0-11e1-9a52-001a4a169705), log id: 657be45b >2012-09-02 18:04:58,509 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (pool-4-thread-50) [2880cbdc] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 3880c572-f4f0-11e1-9a52-001a4a169705 Type: VDS >2012-09-02 18:04:58,511 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (pool-4-thread-50) [34b86514] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 3880c572-f4f0-11e1-9a52-001a4a169705 Type: VDS >2012-09-02 18:04:58,512 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (pool-4-thread-50) [34b86514] FINISH, ActivateVdsVDSCommand, log id: 657be45b >2012-09-02 18:04:58,562 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (pool-4-thread-50) [34b86514] Lock freed to object EngineLock [exclusiveLocks= key: 3880c572-f4f0-11e1-9a52-001a4a169705 value: VDS >, sharedLocks= ] >2012-09-02 18:04:58,955 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-72) [5bea9086] Running command: InitVdsOnUpCommand internal: true. >2012-09-02 18:04:59,756 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-72) [31615828] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: bd560b80-c245-46b3-ad8c-b142a3460cf6 Type: StoragePool >2012-09-02 18:04:59,847 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-72) [31615828] START, ConnectStoragePoolVDSCommand(vdsId = 3880c572-f4f0-11e1-9a52-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 2, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 7530a693 >2012-09-02 18:04:59,871 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-72) [31615828] FINISH, ConnectStoragePoolVDSCommand, log id: 7530a693 >2012-09-02 18:04:59,876 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-72) [31615828] No string for UNASSIGNED type. Use default Log >2012-09-02 18:04:59,879 INFO [org.ovirt.engine.core.bll.MultipleActionsRunner] (pool-4-thread-50) [31615828] MultipleActionsRunner of type MigrateVmToServer invoked with no actions >2012-09-02 18:04:59,897 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-72) [234dcbbe] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 3880c572-f4f0-11e1-9a52-001a4a169705 Type: VDS >2012-09-02 18:04:59,966 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-72) [503c521e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 3880c572-f4f0-11e1-9a52-001a4a169705 Type: VDS >2012-09-02 18:05:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-82) [77190e4a] Autorecovering hosts is disabled, skipping >2012-09-02 18:05:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-82) [77190e4a] Autorecovering storage domains is disabled, skipping >2012-09-02 18:05:05,696 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [5e430da1] hostFromVds::selectedVds - camel-vdsb.qa.lab.tlv.redhat.com, spmStatus Free, storage pool TestDataCenter >2012-09-02 18:05:06,031 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [5e430da1] starting spm on vds camel-vdsb.qa.lab.tlv.redhat.com, storage pool TestDataCenter, prevId -1, LVER 4 >2012-09-02 18:05:06,032 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [5e430da1] START, SpmStartVDSCommand(vdsId = 3880c572-f4f0-11e1-9a52-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, prevId=-1, prevLVER=4, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 421d6f93 >2012-09-02 18:05:06,057 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [5e430da1] spmStart polling started: taskId = fdea7e88-b272-4c5e-9792-99d59cbb2100 >2012-09-02 18:07:16,458 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-85) [5e430da1] Command org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.irsbroker.OneUuidReturnForXmlRpc >mUuid fdea7e88-b272-4c5e-9792-99d59cbb2100 >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 0 >mMessage OK > > >2012-09-02 18:07:16,458 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-85) [5e430da1] Vds: camel-vdsb.qa.lab.tlv.redhat.com >2012-09-02 18:07:16,458 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-85) [5e430da1] Failed in SpmStartVDS method, for vds: camel-vdsb.qa.lab.tlv.redhat.com; host: 10.35.116.2 >2012-09-02 18:07:16,459 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-85) [5e430da1] Command SpmStartVDS execution failed. Exception: NullPointerException: >2012-09-02 18:07:16,459 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [5e430da1] FINISH, SpmStartVDSCommand, log id: 421d6f93 >2012-09-02 18:07:16,463 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-85) [70403ca3] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: bd560b80-c245-46b3-ad8c-b142a3460cf6 Type: StoragePool >2012-09-02 18:07:16,489 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [70403ca3] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-09-02 18:07:16,493 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [70403ca3] Irs placed on server null failed. Proceed Failover >2012-09-02 18:07:16,518 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [70403ca3] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool TestDataCenter >2012-09-02 18:07:16,534 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-85) [70403ca3] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 2d944a13 >2012-09-02 18:07:17,550 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-57) [6af76567] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = 3880c572-f4f0-11e1-9a52-001a4a169705 : camel-vdsb.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 18:07:19,129 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-85) [70403ca3] FINISH, ConnectStoragePoolVDSCommand, log id: 2d944a13 >2012-09-02 18:07:19,329 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [70403ca3] hostFromVds::Connected host to pool - selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Free, storage pool TestDataCenter >2012-09-02 18:07:19,531 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [70403ca3] SpmStatus on vds b32eacf0-f4e9-11e1-973c-001a4a169705: Free >2012-09-02 18:07:19,553 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [70403ca3] starting spm on vds green-vdsa.qa.lab.tlv.redhat.com, storage pool TestDataCenter, prevId 1, LVER 5 >2012-09-02 18:07:19,554 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [70403ca3] START, SpmStartVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, prevId=1, prevLVER=5, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2609fb93 >2012-09-02 18:07:19,558 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-63) [6267f606] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = 3880c572-f4f0-11e1-9a52-001a4a169705 : camel-vdsb.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 18:07:19,579 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [70403ca3] spmStart polling started: taskId = 74244dda-ecaa-448f-9689-dd2769a3f196 >2012-09-02 18:07:21,567 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-22) [632ea3f6] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = 3880c572-f4f0-11e1-9a52-001a4a169705 : camel-vdsb.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 18:07:24,335 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-70) [7fafaa50] Running command: InitVdsOnUpCommand internal: true. >2012-09-02 18:07:24,420 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-70) [53e66ba1] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: bd560b80-c245-46b3-ad8c-b142a3460cf6 Type: StoragePool >2012-09-02 18:07:24,439 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-70) [53e66ba1] START, ConnectStoragePoolVDSCommand(vdsId = 3880c572-f4f0-11e1-9a52-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 2, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 51c6781f >2012-09-02 18:07:26,806 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-70) [53e66ba1] FINISH, ConnectStoragePoolVDSCommand, log id: 51c6781f >2012-09-02 18:07:26,810 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-70) [53e66ba1] No string for UNASSIGNED type. Use default Log >2012-09-02 18:07:26,811 INFO [org.ovirt.engine.core.bll.MultipleActionsRunner] (pool-4-thread-50) [53e66ba1] MultipleActionsRunner of type MigrateVmToServer invoked with no actions >2012-09-02 18:07:26,831 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-70) [f997f4e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 3880c572-f4f0-11e1-9a52-001a4a169705 Type: VDS >2012-09-02 18:07:26,845 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-70) [31a56bd7] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 3880c572-f4f0-11e1-9a52-001a4a169705 Type: VDS >2012-09-02 18:07:41,040 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-85) [70403ca3] Command org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.irsbroker.OneUuidReturnForXmlRpc >mUuid 74244dda-ecaa-448f-9689-dd2769a3f196 >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 0 >mMessage OK > > >2012-09-02 18:07:41,040 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-85) [70403ca3] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 18:07:41,040 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-85) [70403ca3] Failed in SpmStartVDS method, for vds: green-vdsa.qa.lab.tlv.redhat.com; host: 10.35.102.10 >2012-09-02 18:07:41,040 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-85) [70403ca3] Command SpmStartVDS execution failed. Exception: NullPointerException: >2012-09-02 18:07:41,040 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [70403ca3] FINISH, SpmStartVDSCommand, log id: 2609fb93 >2012-09-02 18:07:41,041 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-85) [742ec9a2] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: bd560b80-c245-46b3-ad8c-b142a3460cf6 Type: StoragePool >2012-09-02 18:07:41,446 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [742ec9a2] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-09-02 18:07:42,587 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-100) [1f190ccd] vds::refreshVdsStats Failed getVdsStats, vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, error = VDSNetworkException: VDSNetworkException: >2012-09-02 18:07:42,809 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-100) [1f190ccd] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 18:07:44,819 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-94) [cbe470] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 18:07:46,825 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-87) [49e9877f] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 18:07:48,848 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-6) [23e79838] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@1cd0c3c3 >2012-09-02 18:07:48,848 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-6) [23e79838] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 18:07:48,848 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-6) [23e79838] Command GetCapabilitiesVDS execution failed. Error: VDSRecoveringException: Recovering from crash or Initializing >2012-09-02 18:07:48,877 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-6) [23e79838] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = b32eacf0-f4e9-11e1-973c-001a4a169705 : green-vdsa.qa.lab.tlv.redhat.com, error = Recovering from crash or Initializing, continuing. >2012-09-02 18:07:51,474 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) hostFromVds::selectedVds - camel-vdsb.qa.lab.tlv.redhat.com, spmStatus Free, storage pool TestDataCenter >2012-09-02 18:07:51,495 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) SPM Init: could not find reported vds or not up - pool:TestDataCenter vds_spm_id: 1 >2012-09-02 18:07:51,497 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) SPM selection - vds seems as spm green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 18:07:51,498 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-7) START, SpmStopVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6), log id: 1f344e70 >2012-09-02 18:07:51,518 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-7) SpmStopVDSCommand::Stopping SPM on vds green-vdsa.qa.lab.tlv.redhat.com, pool id bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:07:51,543 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-7) FINISH, SpmStopVDSCommand, log id: 1f344e70 >2012-09-02 18:07:51,543 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) spm stop succeeded, continuing with spm selection >2012-09-02 18:07:51,557 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) starting spm on vds camel-vdsb.qa.lab.tlv.redhat.com, storage pool TestDataCenter, prevId 1, LVER 5 >2012-09-02 18:07:51,558 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) START, SpmStartVDSCommand(vdsId = 3880c572-f4f0-11e1-9a52-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, prevId=1, prevLVER=5, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=true), log id: 6b5427c5 >2012-09-02 18:07:51,586 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) spmStart polling started: taskId = 38e39997-d007-4c36-b72c-eaedcd90cd44 >2012-09-02 18:07:51,587 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-4) [79d32fb4] Cancelling the recovery from crash timer for VDS b32eacf0-f4e9-11e1-973c-001a4a169705 because vds started initializing >2012-09-02 18:07:51,588 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-4) [3808398c] Running command: InitVdsOnUpCommand internal: true. >2012-09-02 18:07:51,692 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-4) [260875b1] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: bd560b80-c245-46b3-ad8c-b142a3460cf6 Type: StoragePool >2012-09-02 18:07:51,708 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-4) [260875b1] START, ConnectStoragePoolVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 1, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 3c85bee6 >2012-09-02 18:07:54,407 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-4) [260875b1] FINISH, ConnectStoragePoolVDSCommand, log id: 3c85bee6 >2012-09-02 18:07:54,412 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-4) [260875b1] No string for UNASSIGNED type. Use default Log >2012-09-02 18:07:54,414 INFO [org.ovirt.engine.core.bll.MultipleActionsRunner] (pool-4-thread-50) [260875b1] MultipleActionsRunner of type MigrateVmToServer invoked with no actions >2012-09-02 18:07:54,455 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-4) [4d0c3ebe] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: b32eacf0-f4e9-11e1-973c-001a4a169705 Type: VDS >2012-09-02 18:07:54,472 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-4) [20d6286c] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: b32eacf0-f4e9-11e1-973c-001a4a169705 Type: VDS >2012-09-02 18:07:54,695 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) spmStart polling ended: taskId = 38e39997-d007-4c36-b72c-eaedcd90cd44 task status = finished >2012-09-02 18:07:54,715 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) spmStart polling ended. spm status: SPM >2012-09-02 18:07:54,717 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-7) START, HSMClearTaskVDSCommand(vdsId = 3880c572-f4f0-11e1-9a52-001a4a169705, taskId=38e39997-d007-4c36-b72c-eaedcd90cd44), log id: 8ced382 >2012-09-02 18:07:54,737 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-7) FINISH, HSMClearTaskVDSCommand, log id: 8ced382 >2012-09-02 18:07:54,737 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@42a00052, log id: 6b5427c5 >2012-09-02 18:07:54,747 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) Initialize Irs proxy from vds: 10.35.116.2 >2012-09-02 18:08:17,288 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8009-2) Checking if user admin@internal is an admin, result true >2012-09-02 18:08:17,288 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8009-2) Running command: LoginUserCommand internal: false. >2012-09-02 18:08:17,367 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8009-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2012-09-02 18:08:17,556 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8009-4) Checking if user admin@internal is an admin, result true >2012-09-02 18:08:17,556 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8009-4) Running command: LoginUserCommand internal: false. >2012-09-02 18:08:17,666 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8009-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2012-09-02 18:08:48,116 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [6ae6e3db] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: camel-vdsb.qa.lab.tlv.redhat.com >2012-09-02 18:09:32,259 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-60) [24ffebc0] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = 3880c572-f4f0-11e1-9a52-001a4a169705 : camel-vdsb.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 18:09:34,267 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-38) [1a3821] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = 3880c572-f4f0-11e1-9a52-001a4a169705 : camel-vdsb.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 18:09:36,274 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-34) [58a0355c] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = 3880c572-f4f0-11e1-9a52-001a4a169705 : camel-vdsb.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 18:09:37,326 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-54) [5637b636] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: bd560b80-c245-46b3-ad8c-b142a3460cf6 Type: StoragePool >2012-09-02 18:09:37,362 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-54) [5637b636] IrsBroker::Failed::GetStoragePoolInfoVDS due to: ConnectException: Connection refused >2012-09-02 18:09:37,365 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-54) [5637b636] Irs placed on server null failed. Proceed Failover >2012-09-02 18:09:37,387 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-54) [5637b636] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Free, storage pool TestDataCenter >2012-09-02 18:09:37,389 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-54) [5637b636] SPM Init: could not find reported vds or not up - pool:TestDataCenter vds_spm_id: 2 >2012-09-02 18:09:37,391 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-54) [5637b636] SPM selection - vds seems as spm camel-vdsb.qa.lab.tlv.redhat.com >2012-09-02 18:09:37,391 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-54) [5637b636] START, SpmStopVDSCommand(vdsId = 3880c572-f4f0-11e1-9a52-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6), log id: 5bb06db0 >2012-09-02 18:09:37,391 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-54) [5637b636] SpmStopVDSCommand:: vds camel-vdsb.qa.lab.tlv.redhat.com is in Connecting status - not performing spm stop, pool id bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:09:37,391 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-54) [5637b636] FINISH, SpmStopVDSCommand, log id: 5bb06db0 >2012-09-02 18:09:37,391 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-54) [5637b636] spm stop on spm failed, stopping spm selection! >2012-09-02 18:09:38,281 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-68) [55173ac5] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = 3880c572-f4f0-11e1-9a52-001a4a169705 : camel-vdsb.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 18:09:40,288 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-50) [5398543e] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = 3880c572-f4f0-11e1-9a52-001a4a169705 : camel-vdsb.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 18:09:42,296 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-73) [3ce1db36] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = 3880c572-f4f0-11e1-9a52-001a4a169705 : camel-vdsb.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 18:09:44,303 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-66) [269bf226] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = 3880c572-f4f0-11e1-9a52-001a4a169705 : camel-vdsb.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 18:09:46,311 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-75) [602f07dc] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = 3880c572-f4f0-11e1-9a52-001a4a169705 : camel-vdsb.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 18:09:47,415 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-76) [3f33ec20] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Free, storage pool TestDataCenter >2012-09-02 18:09:47,417 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-76) [3f33ec20] SPM Init: could not find reported vds or not up - pool:TestDataCenter vds_spm_id: 2 >2012-09-02 18:09:47,418 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-76) [3f33ec20] SPM selection - vds seems as spm camel-vdsb.qa.lab.tlv.redhat.com >2012-09-02 18:09:47,419 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-76) [3f33ec20] START, SpmStopVDSCommand(vdsId = 3880c572-f4f0-11e1-9a52-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6), log id: 4b56f372 >2012-09-02 18:09:47,419 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-76) [3f33ec20] SpmStopVDSCommand:: vds camel-vdsb.qa.lab.tlv.redhat.com is in Connecting status - not performing spm stop, pool id bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:09:47,419 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-76) [3f33ec20] FINISH, SpmStopVDSCommand, log id: 4b56f372 >2012-09-02 18:09:47,419 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-76) [3f33ec20] spm stop on spm failed, stopping spm selection! >2012-09-02 18:09:48,318 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-70) [31a56bd7] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = 3880c572-f4f0-11e1-9a52-001a4a169705 : camel-vdsb.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 18:09:50,326 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-71) [5bcf149a] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = 3880c572-f4f0-11e1-9a52-001a4a169705 : camel-vdsb.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 18:09:52,333 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-90) [5150576a] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = 3880c572-f4f0-11e1-9a52-001a4a169705 : camel-vdsb.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 18:09:54,341 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-72) [503c521e] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = 3880c572-f4f0-11e1-9a52-001a4a169705 : camel-vdsb.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 18:09:56,349 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-92) ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = 3880c572-f4f0-11e1-9a52-001a4a169705 : camel-vdsb.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 18:09:57,441 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-81) [28d52053] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Free, storage pool TestDataCenter >2012-09-02 18:09:57,443 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-81) [28d52053] SPM Init: could not find reported vds or not up - pool:TestDataCenter vds_spm_id: 2 >2012-09-02 18:09:57,445 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-81) [28d52053] SPM selection - vds seems as spm camel-vdsb.qa.lab.tlv.redhat.com >2012-09-02 18:09:57,445 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-81) [28d52053] START, SpmStopVDSCommand(vdsId = 3880c572-f4f0-11e1-9a52-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6), log id: 20df63f3 >2012-09-02 18:09:57,446 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-81) [28d52053] SpmStopVDSCommand:: vds camel-vdsb.qa.lab.tlv.redhat.com is in Connecting status - not performing spm stop, pool id bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:09:57,446 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-81) [28d52053] FINISH, SpmStopVDSCommand, log id: 20df63f3 >2012-09-02 18:09:57,446 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-81) [28d52053] spm stop on spm failed, stopping spm selection! >2012-09-02 18:09:58,356 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-77) [6ae6e3db] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = 3880c572-f4f0-11e1-9a52-001a4a169705 : camel-vdsb.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 18:10:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-95) [20dcf559] Autorecovering hosts is disabled, skipping >2012-09-02 18:10:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-95) [20dcf559] Autorecovering storage domains is disabled, skipping >2012-09-02 18:10:00,363 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-88) [323af124] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = 3880c572-f4f0-11e1-9a52-001a4a169705 : camel-vdsb.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 18:10:02,369 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-100) [1f190ccd] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = 3880c572-f4f0-11e1-9a52-001a4a169705 : camel-vdsb.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 18:10:04,376 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-96) [11d202] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = 3880c572-f4f0-11e1-9a52-001a4a169705 : camel-vdsb.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 18:10:06,383 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-93) [6721d3d3] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = 3880c572-f4f0-11e1-9a52-001a4a169705 : camel-vdsb.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 18:10:07,469 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [49e9877f] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Free, storage pool TestDataCenter >2012-09-02 18:10:07,471 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [49e9877f] SPM Init: could not find reported vds or not up - pool:TestDataCenter vds_spm_id: 2 >2012-09-02 18:10:07,472 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [49e9877f] SPM selection - vds seems as spm camel-vdsb.qa.lab.tlv.redhat.com >2012-09-02 18:10:07,473 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-87) [49e9877f] START, SpmStopVDSCommand(vdsId = 3880c572-f4f0-11e1-9a52-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6), log id: 3c1be8a6 >2012-09-02 18:10:07,473 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-87) [49e9877f] SpmStopVDSCommand:: vds camel-vdsb.qa.lab.tlv.redhat.com is in Connecting status - not performing spm stop, pool id bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:10:07,473 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-87) [49e9877f] FINISH, SpmStopVDSCommand, log id: 3c1be8a6 >2012-09-02 18:10:07,473 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [49e9877f] spm stop on spm failed, stopping spm selection! >2012-09-02 18:10:08,389 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-78) [1081755f] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = 3880c572-f4f0-11e1-9a52-001a4a169705 : camel-vdsb.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 18:10:10,397 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-5) [17d20690] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = 3880c572-f4f0-11e1-9a52-001a4a169705 : camel-vdsb.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 18:10:12,403 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-20) [6f64cdb5] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = 3880c572-f4f0-11e1-9a52-001a4a169705 : camel-vdsb.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 18:10:14,411 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-7) ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = 3880c572-f4f0-11e1-9a52-001a4a169705 : camel-vdsb.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 18:10:16,418 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-13) [188d4eb1] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = 3880c572-f4f0-11e1-9a52-001a4a169705 : camel-vdsb.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 18:10:17,496 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-9) [69982a55] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Free, storage pool TestDataCenter >2012-09-02 18:10:17,498 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-9) [69982a55] SPM Init: could not find reported vds or not up - pool:TestDataCenter vds_spm_id: 2 >2012-09-02 18:10:17,499 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-9) [69982a55] SPM selection - vds seems as spm camel-vdsb.qa.lab.tlv.redhat.com >2012-09-02 18:10:17,500 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-9) [69982a55] START, SpmStopVDSCommand(vdsId = 3880c572-f4f0-11e1-9a52-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6), log id: 1bff2234 >2012-09-02 18:10:17,500 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-9) [69982a55] SpmStopVDSCommand:: vds camel-vdsb.qa.lab.tlv.redhat.com is in Connecting status - not performing spm stop, pool id bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:10:17,500 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-9) [69982a55] FINISH, SpmStopVDSCommand, log id: 1bff2234 >2012-09-02 18:10:17,500 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-9) [69982a55] spm stop on spm failed, stopping spm selection! >2012-09-02 18:10:18,423 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-17) [7445a7b7] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = 3880c572-f4f0-11e1-9a52-001a4a169705 : camel-vdsb.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 18:10:20,430 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-16) [1ca25f7e] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = 3880c572-f4f0-11e1-9a52-001a4a169705 : camel-vdsb.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 18:10:22,447 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-29) ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = 3880c572-f4f0-11e1-9a52-001a4a169705 : camel-vdsb.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 18:10:24,453 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-32) [65b08d21] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = 3880c572-f4f0-11e1-9a52-001a4a169705 : camel-vdsb.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 18:10:26,459 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-28) [1d49a7b8] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = 3880c572-f4f0-11e1-9a52-001a4a169705 : camel-vdsb.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 18:10:27,523 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-40) [31e89cb7] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Free, storage pool TestDataCenter >2012-09-02 18:10:27,525 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-40) [31e89cb7] SPM Init: could not find reported vds or not up - pool:TestDataCenter vds_spm_id: 2 >2012-09-02 18:10:27,526 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-40) [31e89cb7] SPM selection - vds seems as spm camel-vdsb.qa.lab.tlv.redhat.com >2012-09-02 18:10:27,527 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-40) [31e89cb7] START, SpmStopVDSCommand(vdsId = 3880c572-f4f0-11e1-9a52-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6), log id: 451de86 >2012-09-02 18:10:27,527 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-40) [31e89cb7] SpmStopVDSCommand:: vds camel-vdsb.qa.lab.tlv.redhat.com is in Connecting status - not performing spm stop, pool id bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:10:27,527 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-40) [31e89cb7] FINISH, SpmStopVDSCommand, log id: 451de86 >2012-09-02 18:10:27,527 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-40) [31e89cb7] spm stop on spm failed, stopping spm selection! >2012-09-02 18:10:28,465 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-23) [1362cca1] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = 3880c572-f4f0-11e1-9a52-001a4a169705 : camel-vdsb.qa.lab.tlv.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-09-02 18:10:31,189 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-26) [508d81b5] Running command: InitVdsOnUpCommand internal: true. >2012-09-02 18:10:31,242 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (QuartzScheduler_Worker-26) [19b010ce] START, ValidateStorageServerConnectionVDSCommand(vdsId = 3880c572-f4f0-11e1-9a52-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, storageType = ISCSI, connectionList = [{ id: 90cee3d3-f602-49f9-8f17-8665c22f2d06, connection: 10.35.64.10 };]), log id: 15d7fadd >2012-09-02 18:10:31,264 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (QuartzScheduler_Worker-26) [19b010ce] FINISH, ValidateStorageServerConnectionVDSCommand, return: {90cee3d3-f602-49f9-8f17-8665c22f2d06=0}, log id: 15d7fadd >2012-09-02 18:10:31,264 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-26) [19b010ce] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: bd560b80-c245-46b3-ad8c-b142a3460cf6 Type: StoragePool >2012-09-02 18:10:31,265 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (QuartzScheduler_Worker-26) [19b010ce] START, ConnectStorageServerVDSCommand(vdsId = 3880c572-f4f0-11e1-9a52-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, storageType = ISCSI, connectionList = [{ id: 90cee3d3-f602-49f9-8f17-8665c22f2d06, connection: 10.35.64.10 };]), log id: 50ba530a >2012-09-02 18:10:31,343 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (QuartzScheduler_Worker-26) [19b010ce] FINISH, ConnectStorageServerVDSCommand, return: {90cee3d3-f602-49f9-8f17-8665c22f2d06=0}, log id: 50ba530a >2012-09-02 18:10:31,343 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-26) [19b010ce] Host camel-vdsb.qa.lab.tlv.redhat.com storage connection was succeeded >2012-09-02 18:10:31,362 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-26) [19b010ce] START, ConnectStoragePoolVDSCommand(vdsId = 3880c572-f4f0-11e1-9a52-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 2, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 33492283 >2012-09-02 18:10:32,408 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-26) [19b010ce] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 304 >mMessage Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' > > >2012-09-02 18:10:32,409 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-26) [19b010ce] Vds: camel-vdsb.qa.lab.tlv.redhat.com >2012-09-02 18:10:32,409 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-26) [19b010ce] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=bd560b80-c245-46b3-ad8c-b142a3460cf6, msdUUID=56a64ce1-90ab-4576-86fa-c269160aaa38' >2012-09-02 18:10:32,409 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-26) [19b010ce] FINISH, ConnectStoragePoolVDSCommand, log id: 33492283 >2012-09-02 18:10:32,409 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-26) [19b010ce] Could not connect host camel-vdsb.qa.lab.tlv.redhat.com to pool TestDataCenter >2012-09-02 18:10:32,411 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-26) [678e0c3] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 3880c572-f4f0-11e1-9a52-001a4a169705 Type: VDS >2012-09-02 18:10:32,411 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-26) [678e0c3] START, SetVdsStatusVDSCommand(vdsId = 3880c572-f4f0-11e1-9a52-001a4a169705, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 78c6c102 >2012-09-02 18:10:32,423 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-26) [678e0c3] SetVdsStatusVDSCommand::VSD camel-vdsb.qa.lab.tlv.redhat.com is spm and moved from up calling ResetIrs. >2012-09-02 18:10:32,423 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-26) [678e0c3] START, ResetIrsVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = 10.35.116.2, vdsId = 3880c572-f4f0-11e1-9a52-001a4a169705, ignoreStopFailed = false), log id: 30ef85de >2012-09-02 18:10:32,424 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-26) [678e0c3] START, SpmStopVDSCommand(vdsId = 3880c572-f4f0-11e1-9a52-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6), log id: 49fa1e6e >2012-09-02 18:10:32,453 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-26) [678e0c3] SpmStopVDSCommand::Stopping SPM on vds camel-vdsb.qa.lab.tlv.redhat.com, pool id bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:10:32,490 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-26) [678e0c3] FINISH, SpmStopVDSCommand, log id: 49fa1e6e >2012-09-02 18:10:32,491 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-26) [678e0c3] FINISH, ResetIrsVDSCommand, log id: 30ef85de >2012-09-02 18:10:32,491 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-26) [678e0c3] FINISH, SetVdsStatusVDSCommand, log id: 78c6c102 >2012-09-02 18:10:32,576 INFO [org.ovirt.engine.core.bll.MultipleActionsRunner] (pool-4-thread-49) [678e0c3] MultipleActionsRunner of type MigrateVmToServer invoked with no actions >2012-09-02 18:10:32,589 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-26) [7e05cf2e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 3880c572-f4f0-11e1-9a52-001a4a169705 Type: VDS >2012-09-02 18:10:32,602 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-26) [c2cfb36] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 3880c572-f4f0-11e1-9a52-001a4a169705 Type: VDS >2012-09-02 18:10:32,603 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-26) [c2cfb36] Host 3880c572-f4f0-11e1-9a52-001a4a169705 : camel-vdsb.qa.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2012-09-02 18:10:37,550 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [18c0d8a4] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus Free, storage pool TestDataCenter >2012-09-02 18:10:37,576 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [18c0d8a4] SpmStatus on vds 3880c572-f4f0-11e1-9a52-001a4a169705: Unknown_Pool >2012-09-02 18:10:37,590 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [18c0d8a4] starting spm on vds green-vdsa.qa.lab.tlv.redhat.com, storage pool TestDataCenter, prevId 2, LVER 7 >2012-09-02 18:10:37,591 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-39) [18c0d8a4] START, SpmStartVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, prevId=2, prevLVER=7, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 658c9ed5 >2012-09-02 18:10:37,613 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-39) [18c0d8a4] spmStart polling started: taskId = 0e4f76c6-051b-4eb1-9d5b-7a799db786f7 >2012-09-02 18:10:38,649 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-39) [18c0d8a4] Failed in HSMGetTaskStatusVDS method >2012-09-02 18:10:38,649 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-39) [18c0d8a4] Error code GeneralException and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Failed reload: 56a64ce1-90ab-4576-86fa-c269160aaa38 >2012-09-02 18:10:38,649 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-39) [18c0d8a4] spmStart polling ended: taskId = 0e4f76c6-051b-4eb1-9d5b-7a799db786f7 task status = finished >2012-09-02 18:10:38,649 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-39) [18c0d8a4] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Failed reload: 56a64ce1-90ab-4576-86fa-c269160aaa38 >2012-09-02 18:10:38,896 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-39) [18c0d8a4] Failed in SpmStatusVDS method >2012-09-02 18:10:38,903 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-39) [18c0d8a4] Error code GeneralException and error message VDSGenericException: VDSErrorException: Failed to SpmStatusVDS, error = Failed reload: 56a64ce1-90ab-4576-86fa-c269160aaa38 >2012-09-02 18:10:38,903 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-39) [18c0d8a4] Command SpmStatusVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to SpmStatusVDS, error = Failed reload: 56a64ce1-90ab-4576-86fa-c269160aaa38 >2012-09-02 18:10:38,903 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-39) [18c0d8a4] Command org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.irsbroker.OneUuidReturnForXmlRpc >mUuid 0e4f76c6-051b-4eb1-9d5b-7a799db786f7 >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 0 >mMessage OK > > >2012-09-02 18:10:38,904 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-39) [18c0d8a4] Vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 18:10:38,904 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-39) [18c0d8a4] Failed in SpmStartVDS method, for vds: green-vdsa.qa.lab.tlv.redhat.com; host: 10.35.102.10 >2012-09-02 18:10:38,904 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-39) [18c0d8a4] Command SpmStartVDS execution failed. Exception: NullPointerException: >2012-09-02 18:10:38,904 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-39) [18c0d8a4] FINISH, SpmStartVDSCommand, log id: 658c9ed5 >2012-09-02 18:10:38,907 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-39) [5c454588] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: bd560b80-c245-46b3-ad8c-b142a3460cf6 Type: StoragePool >2012-09-02 18:10:38,959 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [5c454588] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-09-02 18:10:49,209 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-50) [5398543e] Failed in SpmStatusVDS method >2012-09-02 18:10:49,209 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-50) [5398543e] Error code GeneralException and error message VDSGenericException: VDSErrorException: Failed to SpmStatusVDS, error = Failed reload: 56a64ce1-90ab-4576-86fa-c269160aaa38 >2012-09-02 18:10:49,209 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-50) [5398543e] Command SpmStatusVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to SpmStatusVDS, error = Failed reload: 56a64ce1-90ab-4576-86fa-c269160aaa38 >2012-09-02 18:10:49,209 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [5398543e] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 18:10:59,459 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-65) [742a9b47] Failed in SpmStatusVDS method >2012-09-02 18:10:59,459 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-65) [742a9b47] Error code GeneralException and error message VDSGenericException: VDSErrorException: Failed to SpmStatusVDS, error = Failed reload: 56a64ce1-90ab-4576-86fa-c269160aaa38 >2012-09-02 18:10:59,459 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-65) [742a9b47] Command SpmStatusVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to SpmStatusVDS, error = Failed reload: 56a64ce1-90ab-4576-86fa-c269160aaa38 >2012-09-02 18:10:59,459 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) [742a9b47] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 18:11:09,719 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-61) [77cb3047] Failed in SpmStatusVDS method >2012-09-02 18:11:09,719 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-61) [77cb3047] Error code GeneralException and error message VDSGenericException: VDSErrorException: Failed to SpmStatusVDS, error = Failed reload: 56a64ce1-90ab-4576-86fa-c269160aaa38 >2012-09-02 18:11:09,719 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-61) [77cb3047] Command SpmStatusVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to SpmStatusVDS, error = Failed reload: 56a64ce1-90ab-4576-86fa-c269160aaa38 >2012-09-02 18:11:09,719 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [77cb3047] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 18:11:19,941 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-7) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=camel-vdsb.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:11:19,979 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-6) [23e79838] Failed in SpmStatusVDS method >2012-09-02 18:11:19,979 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-6) [23e79838] Error code GeneralException and error message VDSGenericException: VDSErrorException: Failed to SpmStatusVDS, error = Failed reload: 56a64ce1-90ab-4576-86fa-c269160aaa38 >2012-09-02 18:11:19,979 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-6) [23e79838] Command SpmStatusVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to SpmStatusVDS, error = Failed reload: 56a64ce1-90ab-4576-86fa-c269160aaa38 >2012-09-02 18:11:19,979 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-6) [23e79838] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 18:11:29,976 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-2) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=camel-vdsb.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:11:30,238 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-11) [28bf429a] Failed in SpmStatusVDS method >2012-09-02 18:11:30,238 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-11) [28bf429a] Error code GeneralException and error message VDSGenericException: VDSErrorException: Failed to SpmStatusVDS, error = Failed reload: 56a64ce1-90ab-4576-86fa-c269160aaa38 >2012-09-02 18:11:30,238 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-11) [28bf429a] Command SpmStatusVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to SpmStatusVDS, error = Failed reload: 56a64ce1-90ab-4576-86fa-c269160aaa38 >2012-09-02 18:11:30,238 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-11) [28bf429a] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 18:11:40,013 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-4) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=camel-vdsb.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:11:40,511 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-25) [24ffe179] Failed in SpmStatusVDS method >2012-09-02 18:11:40,511 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-25) [24ffe179] Error code GeneralException and error message VDSGenericException: VDSErrorException: Failed to SpmStatusVDS, error = Failed reload: 56a64ce1-90ab-4576-86fa-c269160aaa38 >2012-09-02 18:11:40,511 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-25) [24ffe179] Command SpmStatusVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to SpmStatusVDS, error = Failed reload: 56a64ce1-90ab-4576-86fa-c269160aaa38 >2012-09-02 18:11:40,511 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [24ffe179] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 18:11:50,049 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-11) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=camel-vdsb.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:11:50,771 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-46) [69d4155f] Failed in SpmStatusVDS method >2012-09-02 18:11:50,771 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-46) [69d4155f] Error code GeneralException and error message VDSGenericException: VDSErrorException: Failed to SpmStatusVDS, error = Failed reload: 56a64ce1-90ab-4576-86fa-c269160aaa38 >2012-09-02 18:11:50,771 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-46) [69d4155f] Command SpmStatusVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to SpmStatusVDS, error = Failed reload: 56a64ce1-90ab-4576-86fa-c269160aaa38 >2012-09-02 18:11:50,771 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [69d4155f] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 18:12:00,084 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-2) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=camel-vdsb.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:12:01,034 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-55) [65087b1b] Failed in SpmStatusVDS method >2012-09-02 18:12:01,034 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-55) [65087b1b] Error code GeneralException and error message VDSGenericException: VDSErrorException: Failed to SpmStatusVDS, error = Failed reload: 56a64ce1-90ab-4576-86fa-c269160aaa38 >2012-09-02 18:12:01,034 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-55) [65087b1b] Command SpmStatusVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to SpmStatusVDS, error = Failed reload: 56a64ce1-90ab-4576-86fa-c269160aaa38 >2012-09-02 18:12:01,034 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) [65087b1b] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 18:12:10,129 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-9) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=camel-vdsb.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:12:11,295 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-79) Failed in SpmStatusVDS method >2012-09-02 18:12:11,295 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-79) Error code GeneralException and error message VDSGenericException: VDSErrorException: Failed to SpmStatusVDS, error = Failed reload: 56a64ce1-90ab-4576-86fa-c269160aaa38 >2012-09-02 18:12:11,295 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-79) Command SpmStatusVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to SpmStatusVDS, error = Failed reload: 56a64ce1-90ab-4576-86fa-c269160aaa38 >2012-09-02 18:12:11,295 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-79) hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 18:12:20,165 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-10) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=camel-vdsb.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:12:21,556 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-95) [20dcf559] Failed in SpmStatusVDS method >2012-09-02 18:12:21,556 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-95) [20dcf559] Error code GeneralException and error message VDSGenericException: VDSErrorException: Failed to SpmStatusVDS, error = Failed reload: 56a64ce1-90ab-4576-86fa-c269160aaa38 >2012-09-02 18:12:21,556 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-95) [20dcf559] Command SpmStatusVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to SpmStatusVDS, error = Failed reload: 56a64ce1-90ab-4576-86fa-c269160aaa38 >2012-09-02 18:12:21,556 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-95) [20dcf559] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 18:12:30,205 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-4) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=camel-vdsb.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:12:31,823 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-99) [d9b82ea] Failed in SpmStatusVDS method >2012-09-02 18:12:31,823 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-99) [d9b82ea] Error code GeneralException and error message VDSGenericException: VDSErrorException: Failed to SpmStatusVDS, error = Failed reload: 56a64ce1-90ab-4576-86fa-c269160aaa38 >2012-09-02 18:12:31,823 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-99) [d9b82ea] Command SpmStatusVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to SpmStatusVDS, error = Failed reload: 56a64ce1-90ab-4576-86fa-c269160aaa38 >2012-09-02 18:12:31,823 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-99) [d9b82ea] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 18:12:40,233 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-1) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=camel-vdsb.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:12:42,084 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-17) [7445a7b7] Failed in SpmStatusVDS method >2012-09-02 18:12:42,084 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-17) [7445a7b7] Error code GeneralException and error message VDSGenericException: VDSErrorException: Failed to SpmStatusVDS, error = Failed reload: 56a64ce1-90ab-4576-86fa-c269160aaa38 >2012-09-02 18:12:42,085 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-17) [7445a7b7] Command SpmStatusVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to SpmStatusVDS, error = Failed reload: 56a64ce1-90ab-4576-86fa-c269160aaa38 >2012-09-02 18:12:42,085 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [7445a7b7] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 18:12:50,268 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-6) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=camel-vdsb.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:12:52,346 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-41) [727ad312] Failed in SpmStatusVDS method >2012-09-02 18:12:52,346 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-41) [727ad312] Error code GeneralException and error message VDSGenericException: VDSErrorException: Failed to SpmStatusVDS, error = Failed reload: 56a64ce1-90ab-4576-86fa-c269160aaa38 >2012-09-02 18:12:52,346 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-41) [727ad312] Command SpmStatusVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to SpmStatusVDS, error = Failed reload: 56a64ce1-90ab-4576-86fa-c269160aaa38 >2012-09-02 18:12:52,346 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) [727ad312] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 18:13:00,304 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-8) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=camel-vdsb.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:13:02,607 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-46) [69d4155f] Command SpmStatusVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:13:02,607 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [69d4155f] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 18:13:02,607 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [69d4155f] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:13:02,607 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [69d4155f] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:13:03,345 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-46) [4e357f6c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:13:03,395 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-46) [4e357f6c] START, ResetIrsVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, hostName = 10.35.102.10, vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, ignoreStopFailed = true), log id: 7899e3d5 >2012-09-02 18:13:03,396 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-46) [4e357f6c] START, SpmStopVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6), log id: ed6ad41 >2012-09-02 18:13:03,416 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-46) [4e357f6c] SpmStopVDSCommand::Stopping SPM on vds green-vdsa.qa.lab.tlv.redhat.com, pool id bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:13:03,440 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (QuartzScheduler_Worker-46) [4e357f6c] FINISH, SpmStopVDSCommand, log id: ed6ad41 >2012-09-02 18:13:03,468 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (QuartzScheduler_Worker-46) [4e357f6c] FINISH, ResetIrsVDSCommand, log id: 7899e3d5 >2012-09-02 18:13:03,500 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-46) [4e357f6c] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: a584081 >2012-09-02 18:13:03,501 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [4e357f6c] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:13:03,501 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-46) [4e357f6c] FINISH, MarkPoolInReconstructModeVDSCommand, log id: a584081 >2012-09-02 18:13:09,168 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-62) [320e0002] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 18:13:10,333 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-1) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=camel-vdsb.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:13:13,777 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-69) [39fd24f3] Command SpmStatusVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:13:13,777 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [39fd24f3] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 18:13:13,777 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [39fd24f3] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:13:13,778 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [39fd24f3] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:13:14,319 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-69) [54defcc9] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:13:14,392 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-69) [54defcc9] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4c2016af >2012-09-02 18:13:14,392 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [54defcc9] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:13:14,392 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-69) [54defcc9] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4c2016af >2012-09-02 18:13:19,413 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) [742a9b47] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 18:13:20,370 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-7) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=camel-vdsb.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:13:24,659 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-72) [503c521e] Command SpmStatusVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:13:24,659 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [503c521e] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 18:13:24,659 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [503c521e] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:13:24,659 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [503c521e] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:13:24,806 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-72) [5db9ffb3] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:13:24,874 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-72) [5db9ffb3] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1dd3f971 >2012-09-02 18:13:24,874 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [5db9ffb3] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:13:24,874 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-72) [5db9ffb3] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1dd3f971 >2012-09-02 18:13:29,951 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [77cb3047] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 18:13:30,400 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-3) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=camel-vdsb.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:13:35,155 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-3) [3eebd88c] Command SpmStatusVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:13:35,155 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-3) [3eebd88c] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 18:13:35,155 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-3) [3eebd88c] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:13:35,155 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-3) [3eebd88c] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:13:35,733 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-3) [4da23686] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:13:35,781 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-3) [4da23686] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5ef01ced >2012-09-02 18:13:35,781 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-3) [4da23686] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:13:35,781 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-3) [4da23686] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5ef01ced >2012-09-02 18:13:40,435 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-4) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=camel-vdsb.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:13:40,473 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-78) [1081755f] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 18:13:46,055 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-64) [411b2322] Command SpmStatusVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:13:46,055 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-64) [411b2322] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 18:13:46,055 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-64) [411b2322] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:13:46,055 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-64) [411b2322] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:13:46,473 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-64) [7f1d7123] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:13:46,510 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-64) [7f1d7123] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 41053d97 >2012-09-02 18:13:46,510 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-64) [7f1d7123] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:13:46,510 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-64) [7f1d7123] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 41053d97 >2012-09-02 18:13:50,470 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-8) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=camel-vdsb.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:13:50,707 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-19) [66e5caf9] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 18:13:56,795 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-37) [794905fc] Command SpmStatusVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:13:56,795 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-37) [794905fc] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 18:13:56,795 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-37) [794905fc] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:13:56,795 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-37) [794905fc] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:13:56,966 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-37) [5a4adce5] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:13:57,006 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-37) [5a4adce5] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 782c02e3 >2012-09-02 18:13:57,006 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-37) [5a4adce5] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:13:57,006 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-37) [5a4adce5] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 782c02e3 >2012-09-02 18:14:00,505 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-1) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=camel-vdsb.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:14:01,251 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-26) [c2cfb36] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 18:14:07,269 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-39) [5c454588] Command SpmStatusVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:14:07,269 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [5c454588] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 18:14:07,269 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [5c454588] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:14:07,269 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [5c454588] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:14:07,390 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-39) [77f18adc] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:14:07,427 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-39) [77f18adc] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 51de9a3a >2012-09-02 18:14:07,427 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [77f18adc] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:14:07,427 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-39) [77f18adc] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 51de9a3a >2012-09-02 18:14:10,538 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-9) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=camel-vdsb.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:14:11,966 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [4e357f6c] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 18:14:17,702 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-62) [320e0002] Command SpmStatusVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:14:17,702 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-62) [320e0002] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 18:14:17,702 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-62) [320e0002] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:14:17,702 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-62) [320e0002] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:14:17,767 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-62) [59251fd] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:14:17,818 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-62) [59251fd] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 659adb5c >2012-09-02 18:14:17,819 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-62) [59251fd] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:14:17,819 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-62) [59251fd] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 659adb5c >2012-09-02 18:14:20,570 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-7) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=camel-vdsb.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:14:22,165 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [54defcc9] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 18:14:28,087 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-70) [31a56bd7] Command SpmStatusVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:14:28,087 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-70) [31a56bd7] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 18:14:28,087 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-70) [31a56bd7] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:14:28,087 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-70) [31a56bd7] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:14:28,186 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-70) [32a3dc4] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:14:28,230 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-70) [32a3dc4] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1f13a580 >2012-09-02 18:14:28,230 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-70) [32a3dc4] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:14:28,230 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-70) [32a3dc4] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1f13a580 >2012-09-02 18:14:30,605 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-11) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=camel-vdsb.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:14:32,378 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [1571730f] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 18:14:38,520 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-77) [6ae6e3db] Command SpmStatusVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:14:38,520 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [6ae6e3db] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 18:14:38,520 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [6ae6e3db] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:14:38,520 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [6ae6e3db] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:14:38,657 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-77) [bfd9a86] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:14:38,700 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-77) [bfd9a86] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 14084933 >2012-09-02 18:14:38,700 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [bfd9a86] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:14:38,700 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-77) [bfd9a86] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 14084933 >2012-09-02 18:14:40,641 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-4) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=camel-vdsb.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:14:42,590 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-100) [1f190ccd] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 18:14:49,006 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-5) [17d20690] Command SpmStatusVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:14:49,006 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [17d20690] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 18:14:49,006 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [17d20690] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:14:49,006 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [17d20690] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:14:49,092 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-5) [1b411059] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:14:49,147 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-5) [1b411059] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 258027b9 >2012-09-02 18:14:49,147 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [1b411059] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:14:49,147 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-5) [1b411059] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 258027b9 >2012-09-02 18:14:50,686 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-3) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=camel-vdsb.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:14:52,822 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 18:14:59,409 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-15) [5147a4f8] Command SpmStatusVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:14:59,409 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-15) [5147a4f8] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 18:14:59,409 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-15) [5147a4f8] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:14:59,409 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-15) [5147a4f8] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:14:59,492 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-15) [534c1786] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:14:59,528 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-15) [534c1786] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 301d69d3 >2012-09-02 18:14:59,528 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-15) [534c1786] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:14:59,528 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-15) [534c1786] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 301d69d3 >2012-09-02 18:15:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-19) [66e5caf9] Autorecovering hosts is disabled, skipping >2012-09-02 18:15:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-19) [66e5caf9] Autorecovering storage domains is disabled, skipping >2012-09-02 18:15:00,722 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-5) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=camel-vdsb.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:15:03,641 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [37170097] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 18:15:09,786 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-41) [727ad312] Command SpmStatusVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:15:09,786 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) [727ad312] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 18:15:09,787 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) [727ad312] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:15:09,787 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) [727ad312] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:15:09,874 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-41) [5aa958fc] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:15:09,907 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-41) [5aa958fc] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5fc0c9ad >2012-09-02 18:15:09,907 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) [5aa958fc] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:15:09,907 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-41) [5aa958fc] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5fc0c9ad >2012-09-02 18:15:10,757 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-8) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=camel-vdsb.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:15:13,853 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-43) [3ecfc237] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 18:15:20,262 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-46) [4e357f6c] Command SpmStatusVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:15:20,262 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [4e357f6c] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 18:15:20,262 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [4e357f6c] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:15:20,262 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [4e357f6c] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:15:20,611 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-46) [e2ead9e] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:15:20,649 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-46) [e2ead9e] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1c2ab8b8 >2012-09-02 18:15:20,649 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [e2ead9e] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:15:20,649 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-46) [e2ead9e] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1c2ab8b8 >2012-09-02 18:15:20,792 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-2) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=camel-vdsb.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:15:24,152 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-34) [58a0355c] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 18:15:30,827 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-10) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=camel-vdsb.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:15:30,981 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-58) [61be7fc9] Command SpmStatusVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:15:30,981 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-58) [61be7fc9] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 18:15:30,981 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-58) [61be7fc9] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:15:30,981 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-58) [61be7fc9] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:15:31,816 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-58) [3f04b675] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:15:31,847 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-58) [3f04b675] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2fdf56a9 >2012-09-02 18:15:31,847 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-58) [3f04b675] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:15:31,847 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-58) [3f04b675] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2fdf56a9 >2012-09-02 18:15:34,604 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-22) [632ea3f6] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 18:15:40,863 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-6) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=camel-vdsb.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:15:42,501 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-84) [1571730f] Command SpmStatusVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:15:42,502 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [1571730f] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 18:15:42,502 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [1571730f] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:15:42,502 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [1571730f] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:15:44,412 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-84) [38d5807f] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:15:44,454 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-84) [38d5807f] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7ad2cb79 >2012-09-02 18:15:44,454 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [38d5807f] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:15:44,454 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-84) [38d5807f] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7ad2cb79 >2012-09-02 18:15:44,907 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-92) domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 18:15:50,899 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-5) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=camel-vdsb.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:15:54,731 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-94) [cbe470] Command SpmStatusVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:15:54,731 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [cbe470] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 18:15:54,731 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [cbe470] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:15:54,731 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [cbe470] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:15:54,812 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-94) [428400a4] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:15:54,886 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-94) [428400a4] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 14727 >2012-09-02 18:15:54,886 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [428400a4] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:15:54,886 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-94) [428400a4] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 14727 >2012-09-02 18:15:55,102 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [55a6af91] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 18:16:00,935 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-4) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=camel-vdsb.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:16:05,145 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-12) [5064a4c0] Command SpmStatusVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:16:05,145 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-12) [5064a4c0] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 18:16:05,145 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-12) [5064a4c0] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:16:05,145 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-12) [5064a4c0] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:16:05,691 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-12) [499fdac8] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:16:05,758 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-12) [499fdac8] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1c4ff962 >2012-09-02 18:16:05,758 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-12) [499fdac8] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:16:05,758 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-12) [499fdac8] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1c4ff962 >2012-09-02 18:16:10,970 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-1) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=camel-vdsb.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:16:16,032 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-40) [31e89cb7] Command SpmStatusVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:16:16,032 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-40) [31e89cb7] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 18:16:16,032 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-40) [31e89cb7] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:16:16,032 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-40) [31e89cb7] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:16:16,363 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-40) [befedfb] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:16:16,397 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-40) [befedfb] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1fe479b5 >2012-09-02 18:16:16,397 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-40) [befedfb] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:16:16,397 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-40) [befedfb] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1fe479b5 >2012-09-02 18:16:21,005 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-9) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=camel-vdsb.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:16:26,592 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-44) [30ec8757] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 18:16:26,739 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-30) [7097f0e4] Command SpmStatusVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:16:26,739 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [7097f0e4] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 18:16:26,739 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [7097f0e4] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:16:26,739 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [7097f0e4] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:16:26,812 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-30) [28f115a9] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:16:26,850 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-30) [28f115a9] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7b5a50dc >2012-09-02 18:16:26,850 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) [28f115a9] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:16:26,850 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-30) [28f115a9] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7b5a50dc >2012-09-02 18:16:31,041 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-7) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=camel-vdsb.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:16:36,802 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-68) [55173ac5] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 18:16:37,114 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-48) [cde8241] Command SpmStatusVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:16:37,114 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [cde8241] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 18:16:37,115 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [cde8241] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:16:37,115 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [cde8241] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:16:37,435 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-48) [5908d31f] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:16:37,470 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-48) [5908d31f] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6b3aeee0 >2012-09-02 18:16:37,470 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [5908d31f] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:16:37,470 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-48) [5908d31f] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6b3aeee0 >2012-09-02 18:16:41,075 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-3) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=camel-vdsb.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:16:47,037 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) [742a9b47] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 18:16:47,738 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-79) Command SpmStatusVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:16:47,738 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-79) hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 18:16:47,738 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-79) IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:16:47,738 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-79) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:16:47,826 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-79) [5cc43765] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:16:47,857 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-79) [5cc43765] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5479f5a8 >2012-09-02 18:16:47,858 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-79) [5cc43765] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:16:47,858 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-79) [5cc43765] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5479f5a8 >2012-09-02 18:16:51,111 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-10) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=camel-vdsb.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:16:57,238 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [77cb3047] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 18:16:58,124 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-77) [bfd9a86] Command SpmStatusVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:16:58,124 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [bfd9a86] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 18:16:58,124 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [bfd9a86] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:16:58,124 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [bfd9a86] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:16:58,201 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-77) [209ba25c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:16:58,238 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-77) [209ba25c] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7c8ad07d >2012-09-02 18:16:58,238 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [209ba25c] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:16:58,238 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-77) [209ba25c] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7c8ad07d >2012-09-02 18:17:01,146 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-9) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=camel-vdsb.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:17:08,449 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-78) [1081755f] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 18:17:08,526 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-6) [23e79838] Command SpmStatusVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:17:08,527 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-6) [23e79838] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 18:17:08,527 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-6) [23e79838] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:17:08,527 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-6) [23e79838] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:17:08,608 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-6) [bfac455] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:17:08,645 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-6) [bfac455] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4cd87958 >2012-09-02 18:17:08,645 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-6) [bfac455] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:17:08,645 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-6) [bfac455] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4cd87958 >2012-09-02 18:17:11,220 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-11) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=camel-vdsb.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:17:18,683 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [7445a7b7] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 18:17:18,890 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-16) [1ca25f7e] Command SpmStatusVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:17:18,891 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [1ca25f7e] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 18:17:18,891 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [1ca25f7e] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:17:18,891 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [1ca25f7e] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:17:19,038 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-16) [40a18e30] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:17:19,064 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-16) [40a18e30] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6e5187b2 >2012-09-02 18:17:19,065 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [40a18e30] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:17:19,065 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-16) [40a18e30] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6e5187b2 >2012-09-02 18:17:21,255 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-8) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=camel-vdsb.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:17:28,884 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [1362cca1] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 18:17:29,323 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-33) Command SpmStatusVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:17:29,323 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-33) hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 18:17:29,323 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-33) IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:17:29,323 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-33) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:17:29,399 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-33) [405ae04a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:17:29,435 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-33) [405ae04a] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 214cf58f >2012-09-02 18:17:29,435 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-33) [405ae04a] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:17:29,435 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-33) [405ae04a] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 214cf58f >2012-09-02 18:17:31,291 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-4) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=camel-vdsb.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:17:39,092 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) [24ffebc0] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 18:17:39,698 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-38) [1a3821] Command SpmStatusVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:17:39,698 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [1a3821] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 18:17:39,698 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [1a3821] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:17:39,698 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [1a3821] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:17:40,478 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-38) [5c56d03] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:17:40,632 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-38) [5c56d03] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 62cf62c0 >2012-09-02 18:17:40,632 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [5c56d03] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:17:40,632 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-38) [5c56d03] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 62cf62c0 >2012-09-02 18:17:41,327 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-10) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=camel-vdsb.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:17:49,297 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) [65087b1b] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 18:17:50,933 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-69) [54defcc9] Command SpmStatusVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:17:50,933 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [54defcc9] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 18:17:50,933 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [54defcc9] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:17:50,933 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [54defcc9] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:17:51,010 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-69) [60bdc673] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:17:51,047 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-69) [60bdc673] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7f3d8997 >2012-09-02 18:17:51,047 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [60bdc673] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:17:51,047 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-69) [60bdc673] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7f3d8997 >2012-09-02 18:17:51,362 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-9) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=camel-vdsb.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:17:59,501 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-91) [52fa7b4e] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 18:18:01,314 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-90) [5150576a] Command SpmStatusVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:18:01,314 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [5150576a] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 18:18:01,314 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [5150576a] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:18:01,314 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [5150576a] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:18:01,397 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-3) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=camel-vdsb.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:18:01,403 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-90) [397f85d2] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:18:01,461 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-90) [397f85d2] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: c3afb6 >2012-09-02 18:18:01,461 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [397f85d2] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:18:01,461 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-90) [397f85d2] FINISH, MarkPoolInReconstructModeVDSCommand, log id: c3afb6 >2012-09-02 18:18:09,935 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-88) [323af124] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 18:18:11,433 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-8) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=camel-vdsb.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:18:11,726 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-96) [11d202] Command SpmStatusVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:18:11,726 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [11d202] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 18:18:11,726 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [11d202] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:18:11,726 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [11d202] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:18:11,802 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-96) [2a809fc8] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:18:11,839 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-96) [2a809fc8] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7cb9e832 >2012-09-02 18:18:11,839 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [2a809fc8] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:18:11,839 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-96) [2a809fc8] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7cb9e832 >2012-09-02 18:18:20,325 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-4) [20d6286c] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 18:18:21,469 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-2) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=camel-vdsb.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:18:22,103 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-1) [2fbdf05f] Command SpmStatusVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:18:22,103 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [2fbdf05f] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 18:18:22,103 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [2fbdf05f] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:18:22,103 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [2fbdf05f] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:18:22,172 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-1) [34a241af] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:18:22,199 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-1) [34a241af] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1465ab19 >2012-09-02 18:18:22,199 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [34a241af] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:18:22,199 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-1) [34a241af] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1465ab19 >2012-09-02 18:18:31,055 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [61ec3f50] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 18:18:31,504 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-10) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=camel-vdsb.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:18:32,466 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-14) [37170097] Command SpmStatusVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:18:32,466 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [37170097] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 18:18:32,466 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [37170097] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:18:32,466 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [37170097] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:18:32,562 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-14) [123fa08b] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:18:32,599 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-14) [123fa08b] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7003be0a >2012-09-02 18:18:32,599 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [123fa08b] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:18:32,599 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-14) [123fa08b] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7003be0a >2012-09-02 18:18:41,540 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-6) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=camel-vdsb.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:18:41,807 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-36) [765c3ffe] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 18:18:42,868 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-43) [3ecfc237] Command SpmStatusVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:18:42,868 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-43) [3ecfc237] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 18:18:42,868 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-43) [3ecfc237] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:18:42,868 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-43) [3ecfc237] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:18:44,847 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-43) [4a9484b7] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:18:44,903 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-43) [4a9484b7] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7a96d42b >2012-09-02 18:18:44,904 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-43) [4a9484b7] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:18:44,904 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-43) [4a9484b7] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7a96d42b >2012-09-02 18:18:51,575 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-5) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=camel-vdsb.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:18:52,010 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-68) [55173ac5] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 18:18:55,177 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-50) [5398543e] Command SpmStatusVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:18:55,177 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [5398543e] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 18:18:55,177 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [5398543e] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:18:55,177 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [5398543e] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:18:55,470 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-50) [4b231673] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:18:55,506 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-50) [4b231673] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 68dacb6f >2012-09-02 18:18:55,506 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [4b231673] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:18:55,506 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-50) [4b231673] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 68dacb6f >2012-09-02 18:19:01,610 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-4) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=camel-vdsb.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:19:03,795 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [4f91000b] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 18:19:05,772 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-71) [5bcf149a] Command SpmStatusVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:19:05,772 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [5bcf149a] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 18:19:05,772 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [5bcf149a] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:19:05,772 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [5bcf149a] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:19:06,415 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-71) [26e8c53a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:19:06,484 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-71) [26e8c53a] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 430ceb6 >2012-09-02 18:19:06,484 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [26e8c53a] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:19:06,484 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-71) [26e8c53a] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 430ceb6 >2012-09-02 18:19:11,646 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-7) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=camel-vdsb.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:19:14,150 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [209ba25c] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 18:19:16,776 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-96) [2a809fc8] Command SpmStatusVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:19:16,776 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [2a809fc8] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 18:19:16,776 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [2a809fc8] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:19:16,776 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [2a809fc8] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:19:18,081 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-96) [7e3876c6] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:19:18,130 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-96) [7e3876c6] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 559e65f5 >2012-09-02 18:19:18,130 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [7e3876c6] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:19:18,130 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-96) [7e3876c6] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 559e65f5 >2012-09-02 18:19:21,681 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-1) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=camel-vdsb.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:19:24,377 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-6) [bfac455] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 18:19:28,402 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-1) [34a241af] Command SpmStatusVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:19:28,402 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [34a241af] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 18:19:28,402 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [34a241af] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:19:28,402 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [34a241af] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:19:28,589 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-1) [15ef6589] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:19:28,632 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-1) [15ef6589] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: b3e276e >2012-09-02 18:19:28,632 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [15ef6589] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:19:28,632 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-1) [15ef6589] FINISH, MarkPoolInReconstructModeVDSCommand, log id: b3e276e >2012-09-02 18:19:31,715 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-4) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=camel-vdsb.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:19:34,584 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [40a18e30] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 18:19:38,907 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-21) [2eb7fc9d] Command SpmStatusVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:19:38,907 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) [2eb7fc9d] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 18:19:38,907 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) [2eb7fc9d] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:19:38,907 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) [2eb7fc9d] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:19:39,018 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-21) [1299bb25] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:19:39,069 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-21) [1299bb25] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4b93bcad >2012-09-02 18:19:39,069 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) [1299bb25] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:19:39,069 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-21) [1299bb25] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4b93bcad >2012-09-02 18:19:41,747 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-2) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=camel-vdsb.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:19:44,805 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-26) [c2cfb36] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 18:19:49,345 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-43) [4a9484b7] Command SpmStatusVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:19:49,345 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-43) [4a9484b7] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 18:19:49,345 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-43) [4a9484b7] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:19:49,345 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-43) [4a9484b7] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:19:49,428 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-43) [721c6a47] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:19:49,477 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-43) [721c6a47] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2b6b414b >2012-09-02 18:19:49,477 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-43) [721c6a47] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:19:49,477 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-43) [721c6a47] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2b6b414b >2012-09-02 18:19:51,783 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-10) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=camel-vdsb.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:19:54,999 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-52) [570552a] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 18:19:59,746 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-73) [3ce1db36] Command SpmStatusVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:19:59,746 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [3ce1db36] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 18:19:59,746 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [3ce1db36] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:19:59,746 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [3ce1db36] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:19:59,817 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-73) [750b0b87] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:19:59,875 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-73) [750b0b87] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7717d7d0 >2012-09-02 18:19:59,875 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [750b0b87] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:19:59,875 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-73) [750b0b87] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7717d7d0 >2012-09-02 18:20:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-55) [65087b1b] Autorecovering hosts is disabled, skipping >2012-09-02 18:20:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-55) [65087b1b] Autorecovering storage domains is disabled, skipping >2012-09-02 18:20:01,810 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-1) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=camel-vdsb.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:20:05,750 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) [6267f606] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 18:20:10,176 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-91) [52fa7b4e] Command SpmStatusVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:20:10,177 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-91) [52fa7b4e] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 18:20:10,177 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-91) [52fa7b4e] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:20:10,177 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-91) [52fa7b4e] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:20:10,573 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-91) [47ccbe29] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:20:10,621 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-91) [47ccbe29] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 9119867 >2012-09-02 18:20:10,621 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-91) [47ccbe29] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:20:10,621 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-91) [47ccbe29] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 9119867 >2012-09-02 18:20:11,851 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-11) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=camel-vdsb.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:20:15,945 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-89) [1b6b895d] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 18:20:20,893 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-88) [323af124] Command SpmStatusVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:20:20,893 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-88) [323af124] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 18:20:20,893 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-88) [323af124] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:20:20,893 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-88) [323af124] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:20:20,975 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-88) [ae1a9a3] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:20:21,008 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-88) [ae1a9a3] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: a44485f >2012-09-02 18:20:21,008 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-88) [ae1a9a3] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:20:21,008 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-88) [ae1a9a3] FINISH, MarkPoolInReconstructModeVDSCommand, log id: a44485f >2012-09-02 18:20:21,886 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-6) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=camel-vdsb.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:20:26,130 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [49e9877f] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 18:20:31,274 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-20) [6f64cdb5] Command SpmStatusVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:20:31,274 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-20) [6f64cdb5] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 18:20:31,274 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-20) [6f64cdb5] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:20:31,274 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-20) [6f64cdb5] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:20:31,742 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-20) [35fd83fa] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:20:31,800 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-20) [35fd83fa] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2467a2ae >2012-09-02 18:20:31,801 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-20) [35fd83fa] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:20:31,801 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-20) [35fd83fa] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2467a2ae >2012-09-02 18:20:31,921 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-5) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=camel-vdsb.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:20:36,324 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-18) [27b5e70] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 18:20:41,955 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-9) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=camel-vdsb.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:20:42,078 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-27) [61ec3f50] Command SpmStatusVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:20:42,078 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [61ec3f50] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 18:20:42,078 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [61ec3f50] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:20:42,078 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [61ec3f50] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:20:42,148 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-27) [1c4f6f09] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:20:42,181 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-27) [1c4f6f09] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 689c79f5 >2012-09-02 18:20:42,182 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [1c4f6f09] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:20:42,182 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-27) [1c4f6f09] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 689c79f5 >2012-09-02 18:20:46,655 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-35) [6514e28e] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 18:20:51,991 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-8) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=camel-vdsb.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:20:52,444 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-56) Command SpmStatusVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:20:52,444 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-56) hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 18:20:52,444 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-56) IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:20:52,444 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-56) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:20:52,525 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-56) [6a368323] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:20:52,557 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-56) [6a368323] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 34e52bc1 >2012-09-02 18:20:52,558 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-56) [6a368323] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:20:52,558 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-56) [6a368323] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 34e52bc1 >2012-09-02 18:20:56,858 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-49) [2538bc32] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 18:21:02,026 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-6) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=camel-vdsb.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:21:02,824 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-67) [25620c1f] Command SpmStatusVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:21:02,824 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-67) [25620c1f] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 18:21:02,824 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-67) [25620c1f] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:21:02,824 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-67) [25620c1f] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:21:02,959 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-67) [7f4a8311] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:21:02,996 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-67) [7f4a8311] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6afe3a7e >2012-09-02 18:21:02,996 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-67) [7f4a8311] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:21:02,996 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-67) [7f4a8311] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6afe3a7e >2012-09-02 18:21:07,068 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [4b231673] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 18:21:12,061 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-7) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=camel-vdsb.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:21:13,286 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-75) [602f07dc] Command SpmStatusVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:21:13,287 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-75) [602f07dc] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 18:21:13,287 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-75) [602f07dc] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:21:13,287 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-75) [602f07dc] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:21:13,417 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-75) [4fd03e4a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:21:13,479 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-75) [4fd03e4a] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 587e6dd4 >2012-09-02 18:21:13,480 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-75) [4fd03e4a] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:21:13,480 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-75) [4fd03e4a] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 587e6dd4 >2012-09-02 18:21:17,266 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [5db9ffb3] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 18:21:22,095 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-3) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=camel-vdsb.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:21:23,752 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-81) [28d52053] Command SpmStatusVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:21:23,752 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-81) [28d52053] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 18:21:23,752 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-81) [28d52053] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:21:23,752 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-81) [28d52053] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:21:23,845 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-81) [741556f0] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:21:23,889 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-81) [741556f0] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 41d0ac7b >2012-09-02 18:21:23,889 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-81) [741556f0] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:21:23,890 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-81) [741556f0] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 41d0ac7b >2012-09-02 18:21:27,470 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-88) [ae1a9a3] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 18:21:32,130 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-1) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=camel-vdsb.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:21:34,153 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-87) [49e9877f] Command SpmStatusVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:21:34,153 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [49e9877f] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 18:21:34,153 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [49e9877f] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:21:34,153 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [49e9877f] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:21:34,269 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-87) [3e585df] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:21:34,313 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-87) [3e585df] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4c5a908f >2012-09-02 18:21:34,313 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [3e585df] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:21:34,313 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-87) [3e585df] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4c5a908f >2012-09-02 18:21:37,797 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-20) [35fd83fa] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 18:21:42,166 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-4) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=camel-vdsb.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:21:44,608 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-17) [7445a7b7] Command SpmStatusVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:21:44,608 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [7445a7b7] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 18:21:44,608 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [7445a7b7] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:21:44,608 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [7445a7b7] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:21:45,612 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-17) [3f2e402f] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:21:45,659 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-17) [3f2e402f] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2ad5d1b6 >2012-09-02 18:21:45,659 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [3f2e402f] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:21:45,659 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-17) [3f2e402f] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2ad5d1b6 >2012-09-02 18:21:48,036 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [123fa08b] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 18:21:52,201 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-6) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=camel-vdsb.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:21:55,938 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-41) [5aa958fc] Command SpmStatusVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:21:55,938 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) [5aa958fc] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 18:21:55,938 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) [5aa958fc] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:21:55,938 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) [5aa958fc] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:21:56,024 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-41) [30c1a65d] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:21:56,066 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-41) [30c1a65d] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7e124e51 >2012-09-02 18:21:56,066 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) [30c1a65d] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:21:56,066 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-41) [30c1a65d] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7e124e51 >2012-09-02 18:21:58,265 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-56) [6a368323] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 18:22:02,236 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-7) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=camel-vdsb.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:22:06,344 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-52) [570552a] Command SpmStatusVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:22:06,345 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-52) [570552a] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 18:22:06,345 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-52) [570552a] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:22:06,345 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-52) [570552a] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:22:07,368 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-52) [62dc882f] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:22:07,412 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-52) [62dc882f] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7f2e29d9 >2012-09-02 18:22:07,412 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-52) [62dc882f] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:22:07,412 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-52) [62dc882f] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7f2e29d9 >2012-09-02 18:22:08,653 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-34) [58a0355c] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 18:22:12,271 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-9) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=camel-vdsb.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:22:17,688 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-74) [12d00b82] Command SpmStatusVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:22:17,688 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [12d00b82] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 18:22:17,688 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [12d00b82] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:22:17,688 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [12d00b82] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:22:17,763 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-74) [5af80ab0] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:22:17,801 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-74) [5af80ab0] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 65d5293e >2012-09-02 18:22:17,802 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [5af80ab0] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:22:17,802 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-74) [5af80ab0] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 65d5293e >2012-09-02 18:22:18,850 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) [742a9b47] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 18:22:22,307 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-8) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=camel-vdsb.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:22:28,066 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-92) Command SpmStatusVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:22:28,066 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-92) hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 18:22:28,066 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-92) IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:22:28,066 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-92) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:22:28,140 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-92) [1a37db4f] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:22:28,185 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-92) [1a37db4f] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5ddd2e6b >2012-09-02 18:22:28,185 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-92) [1a37db4f] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:22:28,185 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-92) [1a37db4f] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5ddd2e6b >2012-09-02 18:22:29,125 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [77cb3047] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 18:22:32,342 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-11) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=camel-vdsb.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:22:38,448 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-2) [2a8c60b0] Command SpmStatusVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:22:38,448 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) [2a8c60b0] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 18:22:38,448 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) [2a8c60b0] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:22:38,448 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) [2a8c60b0] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:22:38,551 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-2) [4a37a751] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:22:38,599 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-2) [4a37a751] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 69c619b5 >2012-09-02 18:22:38,599 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) [4a37a751] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:22:38,599 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-2) [4a37a751] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 69c619b5 >2012-09-02 18:22:39,338 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-78) [1081755f] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 18:22:42,377 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-6) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=camel-vdsb.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:22:48,856 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-18) [27b5e70] Command SpmStatusVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:22:48,856 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-18) [27b5e70] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 18:22:48,856 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-18) [27b5e70] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:22:48,856 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-18) [27b5e70] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:22:48,978 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-18) [48972315] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:22:49,014 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-18) [48972315] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5b18fcb2 >2012-09-02 18:22:49,014 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-18) [48972315] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:22:49,014 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-18) [48972315] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5b18fcb2 >2012-09-02 18:22:49,581 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-15) [534c1786] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 18:22:52,403 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-10) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=camel-vdsb.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:22:59,409 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-23) [1362cca1] Command SpmStatusVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:22:59,409 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [1362cca1] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 18:22:59,409 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [1362cca1] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:22:59,409 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [1362cca1] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:22:59,474 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-23) [1799e78b] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:22:59,501 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-35) [6514e28e] starting ProcessDomainRecovery for domain 56a64ce1-90ab-4576-86fa-c269160aaa38 >2012-09-02 18:22:59,504 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-35) [6514e28e] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 was reported by all hosts in status UP as problematic. But not moving the Domain to NonOperational. Because of is reconstract now >2012-09-02 18:22:59,507 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-23) [1799e78b] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3ba75b70 >2012-09-02 18:22:59,507 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [1799e78b] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:22:59,507 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-23) [1799e78b] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3ba75b70 >2012-09-02 18:22:59,804 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-26) [c2cfb36] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 18:23:02,448 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-9) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=camel-vdsb.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:23:09,839 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-38) [5c56d03] Command SpmStatusVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:23:09,839 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [5c56d03] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 18:23:09,839 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [5c56d03] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:23:09,839 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [5c56d03] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:23:09,903 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-38) [2fc830dc] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:23:09,935 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [e2ead9e] starting ProcessDomainRecovery for domain 56a64ce1-90ab-4576-86fa-c269160aaa38 >2012-09-02 18:23:09,938 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [e2ead9e] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 was reported by all hosts in status UP as problematic. But not moving the Domain to NonOperational. Because of is reconstract now >2012-09-02 18:23:09,972 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-38) [2fc830dc] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 394d9268 >2012-09-02 18:23:09,972 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [2fc830dc] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:23:09,972 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-38) [2fc830dc] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 394d9268 >2012-09-02 18:23:10,098 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-45) [77216586] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 18:23:12,479 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-8) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=camel-vdsb.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:23:20,257 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-69) [60bdc673] Command SpmStatusVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:23:20,257 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [60bdc673] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 18:23:20,257 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [60bdc673] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:23:20,257 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [60bdc673] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:23:20,325 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [269bf226] starting ProcessDomainRecovery for domain 56a64ce1-90ab-4576-86fa-c269160aaa38 >2012-09-02 18:23:20,329 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [269bf226] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 was reported by all hosts in status UP as problematic. But not moving the Domain to NonOperational. Because of is reconstract now >2012-09-02 18:23:20,731 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-69) [306ceeec] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:23:20,769 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-69) [306ceeec] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 714713e4 >2012-09-02 18:23:20,769 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [306ceeec] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:23:20,769 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-69) [306ceeec] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 714713e4 >2012-09-02 18:23:22,515 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-2) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=camel-vdsb.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:23:30,895 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [397f85d2] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 18:23:31,047 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-86) [4effdff3] Command SpmStatusVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:23:31,047 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) [4effdff3] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 18:23:31,048 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) [4effdff3] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:23:31,048 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) [4effdff3] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:23:31,055 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [77190e4a] starting ProcessDomainRecovery for domain 56a64ce1-90ab-4576-86fa-c269160aaa38 >2012-09-02 18:23:31,058 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [77190e4a] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 was reported by all hosts in status UP as problematic. But not moving the Domain to NonOperational. Because of is reconstract now >2012-09-02 18:23:31,145 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-86) [4baa6cc3] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:23:31,184 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-86) [4baa6cc3] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: f6e2096 >2012-09-02 18:23:31,184 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) [4baa6cc3] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:23:31,184 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-86) [4baa6cc3] FINISH, MarkPoolInReconstructModeVDSCommand, log id: f6e2096 >2012-09-02 18:23:32,550 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-4) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=camel-vdsb.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:23:41,123 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [1fab286a] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 18:23:41,459 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-96) [7e3876c6] Command SpmStatusVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:23:41,459 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [7e3876c6] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 18:23:41,459 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [7e3876c6] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:23:41,459 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [7e3876c6] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:23:41,525 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-96) [5d1c4f2a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:23:41,554 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-96) [5d1c4f2a] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3342bbe3 >2012-09-02 18:23:41,554 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [5d1c4f2a] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:23:41,554 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-96) [5d1c4f2a] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3342bbe3 >2012-09-02 18:23:42,577 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-3) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=camel-vdsb.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:23:51,361 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) [7fa2eb8b] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 18:23:51,833 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-12) [499fdac8] Command SpmStatusVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:23:51,834 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-12) [499fdac8] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 18:23:51,834 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-12) [499fdac8] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:23:51,834 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-12) [499fdac8] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:23:51,919 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-12) [129811a3] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:23:51,955 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-12) [129811a3] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7485cf7b >2012-09-02 18:23:51,955 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-12) [129811a3] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:23:51,955 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-12) [129811a3] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7485cf7b >2012-09-02 18:23:52,612 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-10) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=camel-vdsb.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:24:01,593 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [24ffe179] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 18:24:02,210 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-40) [befedfb] Command SpmStatusVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:24:02,210 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-40) [befedfb] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 18:24:02,210 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-40) [befedfb] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:24:02,211 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-40) [befedfb] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:24:02,647 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-7) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=camel-vdsb.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:24:03,099 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-40) [17ceb1f1] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:24:03,134 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-40) [17ceb1f1] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 40e29eaf >2012-09-02 18:24:03,134 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-40) [17ceb1f1] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:24:03,134 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-40) [17ceb1f1] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 40e29eaf >2012-09-02 18:24:11,858 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [77f18adc] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 18:24:12,682 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-6) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=camel-vdsb.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:24:13,393 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-60) [24ffebc0] Command SpmStatusVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:24:13,393 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) [24ffebc0] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 18:24:13,393 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) [24ffebc0] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:24:13,393 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) [24ffebc0] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:24:13,919 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-60) [7b630c03] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:24:14,006 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-60) [7b630c03] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 30515da7 >2012-09-02 18:24:14,006 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) [7b630c03] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:24:14,006 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-60) [7b630c03] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 30515da7 >2012-09-02 18:24:22,084 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [750b0b87] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 18:24:22,718 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-3) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=camel-vdsb.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:24:24,275 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-50) [4b231673] Command SpmStatusVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:24:24,275 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [4b231673] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 18:24:24,275 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [4b231673] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:24:24,275 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [4b231673] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:24:24,342 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-50) [5076b6ae] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:24:24,377 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-50) [5076b6ae] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1e792d3e >2012-09-02 18:24:24,377 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [5076b6ae] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:24:24,377 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-50) [5076b6ae] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1e792d3e >2012-09-02 18:24:32,293 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-79) [5cc43765] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 18:24:32,755 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-9) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=camel-vdsb.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:24:34,584 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [397f85d2] starting ProcessDomainRecovery for domain 56a64ce1-90ab-4576-86fa-c269160aaa38 >2012-09-02 18:24:34,641 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-71) [26e8c53a] Command SpmStatusVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:24:34,641 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [26e8c53a] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 18:24:34,641 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [26e8c53a] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:24:34,641 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [26e8c53a] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:24:34,959 WARN [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (QuartzScheduler_Worker-90) [3c7d51b2] The message key HandleFailedStorageDomain is missing from bundles/ExecutionMessages >2012-09-02 18:24:35,010 INFO [org.ovirt.engine.core.bll.storage.HandleFailedStorageDomainCommand] (QuartzScheduler_Worker-90) [3c7d51b2] Running command: HandleFailedStorageDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:24:35,023 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-90) [45273507] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:24:35,121 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (QuartzScheduler_Worker-90) [45273507] START, DisconnectStorageServerVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, storageType = ISCSI, connectionList = [{ id: 90cee3d3-f602-49f9-8f17-8665c22f2d06, connection: 10.35.64.10 };]), log id: 44674aee >2012-09-02 18:24:35,963 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (QuartzScheduler_Worker-90) [45273507] FINISH, DisconnectStorageServerVDSCommand, return: {90cee3d3-f602-49f9-8f17-8665c22f2d06=0}, log id: 44674aee >2012-09-02 18:24:35,963 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-90) [45273507] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2a08463a >2012-09-02 18:24:35,963 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [45273507] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:24:35,963 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-90) [45273507] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2a08463a >2012-09-02 18:24:35,977 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-90) [45273507] No string for UNASSIGNED type. Use default Log >2012-09-02 18:24:42,791 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-8) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=camel-vdsb.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:24:44,865 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-96) [5d1c4f2a] Command SpmStatusVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:24:44,865 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [5d1c4f2a] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 18:24:44,865 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [5d1c4f2a] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:24:44,865 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [5d1c4f2a] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:24:44,946 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-96) [61696304] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:24:44,983 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-96) [61696304] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7696342c >2012-09-02 18:24:44,983 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [61696304] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:24:44,983 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-96) [61696304] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7696342c >2012-09-02 18:24:52,826 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-2) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=camel-vdsb.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:24:55,207 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-13) [188d4eb1] Command SpmStatusVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:24:55,207 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) [188d4eb1] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 18:24:55,207 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) [188d4eb1] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:24:55,207 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) [188d4eb1] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:24:55,495 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-13) [4541eefc] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:24:55,547 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-13) [4541eefc] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2538459b >2012-09-02 18:24:55,547 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) [4541eefc] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:24:55,547 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-13) [4541eefc] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2538459b >2012-09-02 18:25:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-17) [3f2e402f] Autorecovering hosts is disabled, skipping >2012-09-02 18:25:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-17) [3f2e402f] Autorecovering storage domains is disabled, skipping >2012-09-02 18:25:02,859 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-7) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=camel-vdsb.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:25:05,810 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-21) [1299bb25] Command SpmStatusVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:25:05,810 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) [1299bb25] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 18:25:05,810 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) [1299bb25] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:25:05,810 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) [1299bb25] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:25:06,454 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-21) [5518a685] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:25:06,508 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-21) [5518a685] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2bf6947f >2012-09-02 18:25:06,509 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) [5518a685] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:25:06,509 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-21) [5518a685] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2bf6947f >2012-09-02 18:25:12,884 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-10) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=camel-vdsb.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:25:16,726 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-49) [2538bc32] Command SpmStatusVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:25:16,726 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-49) [2538bc32] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 18:25:16,726 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-49) [2538bc32] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:25:16,726 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-49) [2538bc32] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:25:16,860 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-49) [4075356a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:25:16,896 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-49) [4075356a] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 588f01bd >2012-09-02 18:25:16,896 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-49) [4075356a] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:25:16,896 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-49) [4075356a] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 588f01bd >2012-09-02 18:25:22,920 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-3) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=camel-vdsb.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:25:27,125 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-55) [65087b1b] Command SpmStatusVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:25:27,126 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) [65087b1b] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 18:25:27,126 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) [65087b1b] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:25:27,126 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) [65087b1b] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:25:27,222 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-55) [3f271475] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:25:27,259 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-55) [3f271475] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 70580df3 >2012-09-02 18:25:27,259 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) [3f271475] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:25:27,259 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-55) [3f271475] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 70580df3 >2012-09-02 18:25:32,954 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-4) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=camel-vdsb.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:25:37,478 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-91) [47ccbe29] Command SpmStatusVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:25:37,478 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-91) [47ccbe29] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 18:25:37,478 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-91) [47ccbe29] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:25:37,478 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-91) [47ccbe29] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:25:37,590 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-91) [70f89c76] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:25:37,640 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-91) [70f89c76] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 387bfc1d >2012-09-02 18:25:37,640 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-91) [70f89c76] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:25:37,640 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-91) [70f89c76] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 387bfc1d >2012-09-02 18:25:42,996 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-6) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=camel-vdsb.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:25:47,859 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-100) [1f190ccd] Command SpmStatusVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:25:47,859 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-100) [1f190ccd] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 18:25:47,860 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-100) [1f190ccd] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:25:47,860 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-100) [1f190ccd] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:25:47,950 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-100) [9b60e1c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:25:47,988 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-100) [9b60e1c] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 46cf9f1d >2012-09-02 18:25:47,988 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-100) [9b60e1c] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:25:47,988 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-100) [9b60e1c] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 46cf9f1d >2012-09-02 18:25:53,035 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-10) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=camel-vdsb.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:25:58,210 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-7) Command SpmStatusVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:25:58,210 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 18:25:58,210 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:25:58,210 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:25:58,298 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-7) [64c78fe6] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:25:58,328 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-7) [64c78fe6] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 50102531 >2012-09-02 18:25:58,328 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [64c78fe6] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:25:58,328 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-7) [64c78fe6] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 50102531 >2012-09-02 18:26:03,074 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-11) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=camel-vdsb.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:26:08,560 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-32) [65b08d21] Command SpmStatusVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:26:08,560 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-32) [65b08d21] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 18:26:08,560 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-32) [65b08d21] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:26:08,560 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-32) [65b08d21] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:26:08,635 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-32) [22853e9b] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:26:08,675 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-32) [22853e9b] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 95ed1aa >2012-09-02 18:26:08,675 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-32) [22853e9b] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:26:08,675 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-32) [22853e9b] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 95ed1aa >2012-09-02 18:26:13,099 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-8) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=camel-vdsb.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:26:18,893 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-36) [765c3ffe] Command SpmStatusVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:26:18,893 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-36) [765c3ffe] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 18:26:18,894 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-36) [765c3ffe] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:26:18,894 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-36) [765c3ffe] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:26:19,184 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-36) [68598e88] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:26:19,230 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-36) [68598e88] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3342dace >2012-09-02 18:26:19,230 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-36) [68598e88] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:26:19,230 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-36) [68598e88] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3342dace >2012-09-02 18:26:23,127 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-2) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=camel-vdsb.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:26:29,450 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-67) [7f4a8311] Command SpmStatusVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:26:29,450 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-67) [7f4a8311] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 18:26:29,450 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-67) [7f4a8311] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:26:29,450 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-67) [7f4a8311] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:26:29,534 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-67) [7525b7f] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:26:29,584 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-67) [7525b7f] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 313e6457 >2012-09-02 18:26:29,584 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-67) [7525b7f] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:26:29,584 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-67) [7525b7f] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 313e6457 >2012-09-02 18:26:33,168 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-10) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=camel-vdsb.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:26:39,812 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-75) [4fd03e4a] Command SpmStatusVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:26:39,812 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-75) [4fd03e4a] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 18:26:39,812 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-75) [4fd03e4a] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:26:39,812 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-75) [4fd03e4a] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:26:40,001 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-75) [31c89035] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:26:40,047 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-75) [31c89035] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1b9ee068 >2012-09-02 18:26:40,047 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-75) [31c89035] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:26:40,047 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-75) [31c89035] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1b9ee068 >2012-09-02 18:26:43,204 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-8) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=camel-vdsb.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:26:50,277 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-77) [209ba25c] Command SpmStatusVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:26:50,278 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [209ba25c] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 18:26:50,278 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [209ba25c] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:26:50,278 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [209ba25c] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:26:50,569 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-77) [5c6e7843] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:26:50,662 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-77) [5c6e7843] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 35c118b4 >2012-09-02 18:26:50,662 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [5c6e7843] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:26:50,662 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-77) [5c6e7843] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 35c118b4 >2012-09-02 18:26:53,242 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-9) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=camel-vdsb.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:27:00,935 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-6) [bfac455] Command SpmStatusVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:27:00,935 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-6) [bfac455] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 18:27:00,935 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-6) [bfac455] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:27:00,935 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-6) [bfac455] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:27:02,326 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-6) [7c760cba] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:27:02,396 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-6) [7c760cba] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 26db85f0 >2012-09-02 18:27:02,396 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-6) [7c760cba] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:27:02,396 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-6) [7c760cba] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 26db85f0 >2012-09-02 18:27:03,278 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-6) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=camel-vdsb.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:27:12,615 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-16) [40a18e30] Command SpmStatusVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:27:12,615 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [40a18e30] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 18:27:12,615 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [40a18e30] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:27:12,615 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [40a18e30] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:27:12,694 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-16) [1cdd254a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:27:12,727 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-16) [1cdd254a] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5925d2d1 >2012-09-02 18:27:12,727 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [1cdd254a] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:27:12,727 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-16) [1cdd254a] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5925d2d1 >2012-09-02 18:27:13,329 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-11) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=camel-vdsb.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:27:22,948 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-41) [30c1a65d] Command SpmStatusVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:27:22,948 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) [30c1a65d] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 18:27:22,948 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) [30c1a65d] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:27:22,948 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) [30c1a65d] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:27:23,020 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-41) [5b5408a2] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:27:23,049 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-41) [5b5408a2] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2720f339 >2012-09-02 18:27:23,049 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) [5b5408a2] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:27:23,049 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-41) [5b5408a2] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2720f339 >2012-09-02 18:27:23,369 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-10) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=camel-vdsb.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:27:33,270 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-47) [9d34e64] Command SpmStatusVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:27:33,270 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-47) [9d34e64] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 18:27:33,271 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-47) [9d34e64] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:27:33,271 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-47) [9d34e64] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:27:33,369 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-47) [24e2af1f] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:27:33,402 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-9) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=camel-vdsb.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:27:33,413 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-47) [24e2af1f] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1971eb8e >2012-09-02 18:27:33,413 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-47) [24e2af1f] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:27:33,413 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-47) [24e2af1f] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1971eb8e >2012-09-02 18:27:43,449 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-4) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=camel-vdsb.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:27:43,639 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-22) [632ea3f6] Command SpmStatusVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:27:43,640 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-22) [632ea3f6] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 18:27:43,640 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-22) [632ea3f6] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:27:43,640 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-22) [632ea3f6] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:27:43,899 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-22) [452f8136] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:27:43,945 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-22) [452f8136] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 623e83ad >2012-09-02 18:27:43,945 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-22) [452f8136] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:27:43,945 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-22) [452f8136] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 623e83ad >2012-09-02 18:27:53,485 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-3) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=camel-vdsb.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:27:54,240 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-84) [38d5807f] Command SpmStatusVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:27:54,241 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [38d5807f] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 18:27:54,241 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [38d5807f] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:27:54,241 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [38d5807f] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:27:54,310 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-84) [42ba8224] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:27:54,336 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-84) [42ba8224] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4ff4d710 >2012-09-02 18:27:54,336 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [42ba8224] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:27:54,336 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-84) [42ba8224] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4ff4d710 >2012-09-02 18:28:03,519 WARN [org.ovirt.engine.core.bll.SearchQuery] (ajp-/127.0.0.1:8009-10) ResourceManager::searchBusinessObjects - erroneous search text - Hosts : name=camel-vdsb.qa.lab.tlv.redhat.com and status=non_operational >2012-09-02 18:28:04,642 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-2) [4a37a751] Command SpmStatusVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:28:04,642 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) [4a37a751] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 18:28:04,642 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) [4a37a751] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:28:04,642 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) [4a37a751] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:28:04,739 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-2) [59d1e4ca] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:28:04,776 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-2) [59d1e4ca] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 72a25c0d >2012-09-02 18:28:04,776 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) [59d1e4ca] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:28:04,776 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-2) [59d1e4ca] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 72a25c0d >2012-09-02 18:28:15,242 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-9) [69982a55] Command SpmStatusVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:28:15,242 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-9) [69982a55] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 18:28:15,242 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-9) [69982a55] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:28:15,242 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-9) [69982a55] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:28:16,330 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-9) [5d8d8d2c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:28:16,392 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-9) [5d8d8d2c] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 792161c0 >2012-09-02 18:28:16,392 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-9) [5d8d8d2c] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:28:16,392 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-9) [5d8d8d2c] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 792161c0 >2012-09-02 18:28:26,610 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-23) [1799e78b] Command SpmStatusVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:28:26,610 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [1799e78b] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 18:28:26,610 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [1799e78b] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:28:26,610 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [1799e78b] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:28:26,681 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-23) [3fd9ed84] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:28:26,711 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-23) [3fd9ed84] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: a44665c >2012-09-02 18:28:26,711 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [3fd9ed84] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:28:26,711 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-23) [3fd9ed84] FINISH, MarkPoolInReconstructModeVDSCommand, log id: a44665c >2012-09-02 18:28:34,726 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (ajp-/127.0.0.1:8009-1) [e249b58] Lock Acquired to object EngineLock [exclusiveLocks= key: 3880c572-f4f0-11e1-9a52-001a4a169705 value: VDS >, sharedLocks= ] >2012-09-02 18:28:34,748 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (ajp-/127.0.0.1:8009-1) [e249b58] Running command: ActivateVdsCommand internal: false. Entities affected : ID: 3880c572-f4f0-11e1-9a52-001a4a169705 Type: VDS >2012-09-02 18:28:34,749 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp-/127.0.0.1:8009-1) [e249b58] START, SetVdsStatusVDSCommand(vdsId = 3880c572-f4f0-11e1-9a52-001a4a169705, status=Unassigned, nonOperationalReason=NONE), log id: 3ebcfaf3 >2012-09-02 18:28:34,751 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp-/127.0.0.1:8009-1) [e249b58] FINISH, SetVdsStatusVDSCommand, log id: 3ebcfaf3 >2012-09-02 18:28:34,757 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (ajp-/127.0.0.1:8009-1) [e249b58] START, ActivateVdsVDSCommand(vdsId = 3880c572-f4f0-11e1-9a52-001a4a169705), log id: 1567711b >2012-09-02 18:28:35,082 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (ajp-/127.0.0.1:8009-1) [781f4746] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 3880c572-f4f0-11e1-9a52-001a4a169705 Type: VDS >2012-09-02 18:28:35,083 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (ajp-/127.0.0.1:8009-1) [41a4886e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 3880c572-f4f0-11e1-9a52-001a4a169705 Type: VDS >2012-09-02 18:28:35,084 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (ajp-/127.0.0.1:8009-1) [41a4886e] FINISH, ActivateVdsVDSCommand, log id: 1567711b >2012-09-02 18:28:35,543 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (ajp-/127.0.0.1:8009-1) [41a4886e] Lock freed to object EngineLock [exclusiveLocks= key: 3880c572-f4f0-11e1-9a52-001a4a169705 value: VDS >, sharedLocks= ] >2012-09-02 18:28:36,179 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-60) [34ec38b4] Running command: InitVdsOnUpCommand internal: true. >2012-09-02 18:28:36,220 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-60) [247e0d95] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: bd560b80-c245-46b3-ad8c-b142a3460cf6 Type: StoragePool >2012-09-02 18:28:36,240 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-60) [247e0d95] START, ConnectStoragePoolVDSCommand(vdsId = 3880c572-f4f0-11e1-9a52-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, vds_spm_id = 2, masterDomainId = 56a64ce1-90ab-4576-86fa-c269160aaa38, masterVersion = 1), log id: 40459604 >2012-09-02 18:28:36,928 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-46) [e2ead9e] Command SpmStatusVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:28:36,928 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [e2ead9e] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 18:28:36,928 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [e2ead9e] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:28:36,928 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [e2ead9e] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:28:37,081 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-46) [60d6d42d] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 56a64ce1-90ab-4576-86fa-c269160aaa38 Type: Storage >2012-09-02 18:28:37,118 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-46) [60d6d42d] START, MarkPoolInReconstructModeVDSCommand(storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 31222c75 >2012-09-02 18:28:37,118 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [60d6d42d] clearing cache for problematic entities in pool bd560b80-c245-46b3-ad8c-b142a3460cf6 >2012-09-02 18:28:37,119 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-46) [60d6d42d] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 31222c75 >2012-09-02 18:28:38,634 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-60) [247e0d95] FINISH, ConnectStoragePoolVDSCommand, log id: 40459604 >2012-09-02 18:28:38,638 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-60) [247e0d95] No string for UNASSIGNED type. Use default Log >2012-09-02 18:28:38,639 INFO [org.ovirt.engine.core.bll.MultipleActionsRunner] (pool-4-thread-47) [247e0d95] MultipleActionsRunner of type MigrateVmToServer invoked with no actions >2012-09-02 18:28:38,654 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-60) [60f6c0b0] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 3880c572-f4f0-11e1-9a52-001a4a169705 Type: VDS >2012-09-02 18:28:38,673 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-60) [7821debc] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 3880c572-f4f0-11e1-9a52-001a4a169705 Type: VDS >2012-09-02 18:28:47,351 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-50) [5076b6ae] Command SpmStatusVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:28:47,351 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [5076b6ae] hostFromVds::selectedVds - green-vdsa.qa.lab.tlv.redhat.com, spmStatus returned null! >2012-09-02 18:28:47,351 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [5076b6ae] IrsBroker::Failed::GetStoragePoolInfoVDS >2012-09-02 18:28:47,351 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [5076b6ae] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:28:47,571 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-50) [5076b6ae] Command SpmStatusVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) >2012-09-02 18:28:47,571 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [5076b6ae] Irs placed on server null failed. Proceed Failover >2012-09-02 18:28:47,839 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [5076b6ae] hostFromVds::selectedVds - camel-vdsb.qa.lab.tlv.redhat.com, spmStatus Free, storage pool TestDataCenter >2012-09-02 18:28:48,073 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [5076b6ae] SpmStatus on vds 3880c572-f4f0-11e1-9a52-001a4a169705: Free >2012-09-02 18:28:48,083 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [5076b6ae] starting spm on vds camel-vdsb.qa.lab.tlv.redhat.com, storage pool TestDataCenter, prevId 2, LVER 7 >2012-09-02 18:28:48,084 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-50) [5076b6ae] START, SpmStartVDSCommand(vdsId = 3880c572-f4f0-11e1-9a52-001a4a169705, storagePoolId = bd560b80-c245-46b3-ad8c-b142a3460cf6, prevId=2, prevLVER=7, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5c17c696 >2012-09-02 18:28:48,110 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-50) [5076b6ae] spmStart polling started: taskId = 0f47fd75-33b4-491d-b4a6-daccbf3a86b8 >2012-09-02 18:29:01,499 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-50) [5076b6ae] spmStart polling ended: taskId = 0f47fd75-33b4-491d-b4a6-daccbf3a86b8 task status = finished >2012-09-02 18:29:01,519 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-50) [5076b6ae] spmStart polling ended. spm status: SPM >2012-09-02 18:29:01,520 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-50) [5076b6ae] START, HSMClearTaskVDSCommand(vdsId = 3880c572-f4f0-11e1-9a52-001a4a169705, taskId=0f47fd75-33b4-491d-b4a6-daccbf3a86b8), log id: 107cac4a >2012-09-02 18:29:01,541 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-50) [5076b6ae] FINISH, HSMClearTaskVDSCommand, log id: 107cac4a >2012-09-02 18:29:01,541 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-50) [5076b6ae] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@691336ba, log id: 5c17c696 >2012-09-02 18:29:01,599 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [5076b6ae] Initialize Irs proxy from vds: 10.35.116.2 >2012-09-02 18:29:01,980 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [77cb3047] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 18:29:11,858 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [3e585df] starting ProcessDomainRecovery for domain 56a64ce1-90ab-4576-86fa-c269160aaa38 >2012-09-02 18:29:11,862 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [3e585df] vds green-vdsa.qa.lab.tlv.redhat.com reported domain 56a64ce1-90ab-4576-86fa-c269160aaa38:ISCSIDataDomain as in problem, moving the vds to status NonOperational >2012-09-02 18:29:11,865 WARN [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (QuartzScheduler_Worker-87) [133f9b] The message key SetNonOperationalVds is missing from bundles/ExecutionMessages >2012-09-02 18:29:12,601 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-87) [133f9b] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: b32eacf0-f4e9-11e1-973c-001a4a169705 Type: VDS >2012-09-02 18:29:12,601 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-87) [133f9b] START, SetVdsStatusVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 52652e7b >2012-09-02 18:29:12,663 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-87) [133f9b] FINISH, SetVdsStatusVDSCommand, log id: 52652e7b >2012-09-02 18:29:12,724 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [133f9b] Clearing cache of pool: bd560b80-c245-46b3-ad8c-b142a3460cf6 for problematic entities of VDS: green-vdsa.qa.lab.tlv.redhat.com. >2012-09-02 18:29:12,724 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [133f9b] Domain 56a64ce1-90ab-4576-86fa-c269160aaa38 recovered from problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 18:29:12,724 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [133f9b] Domain 56a64ce1-90ab-4576-86fa-c269160aaa38 has recovered from problem. No active host in the DC is reporting it as problematic, so clearing the domain recovery timer. >2012-09-02 18:29:12,724 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-99) [d9b82ea] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 18:29:12,774 INFO [org.ovirt.engine.core.bll.InternalMigrateVmCommand] (pool-4-thread-47) [5a60a9a5] Running command: InternalMigrateVmCommand internal: true. Entities affected : ID: 5449d6cf-0e31-44b5-b0a9-65036f122662 Type: VM >2012-09-02 18:29:12,780 INFO [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand] (pool-4-thread-47) [5a60a9a5] START, MigrateVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, vmId=5449d6cf-0e31-44b5-b0a9-65036f122662, srcHost=10.35.102.10, dstVdsId=3880c572-f4f0-11e1-9a52-001a4a169705, dstHost=10.35.116.2:54321, migrationMethod=ONLINE), log id: 4282c86d >2012-09-02 18:29:12,781 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] (pool-4-thread-47) [5a60a9a5] VdsBroker::migrate::Entered (vm_guid=5449d6cf-0e31-44b5-b0a9-65036f122662, srcHost=10.35.102.10, dstHost=10.35.116.2:54321, method=online >2012-09-02 18:29:12,781 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] (pool-4-thread-47) [5a60a9a5] START, MigrateBrokerVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, vmId=5449d6cf-0e31-44b5-b0a9-65036f122662, srcHost=10.35.102.10, dstVdsId=3880c572-f4f0-11e1-9a52-001a4a169705, dstHost=10.35.116.2:54321, migrationMethod=ONLINE), log id: 106c53f5 >2012-09-02 18:29:12,800 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] (pool-4-thread-47) [5a60a9a5] FINISH, MigrateBrokerVDSCommand, log id: 106c53f5 >2012-09-02 18:29:12,815 INFO [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand] (pool-4-thread-47) [5a60a9a5] FINISH, MigrateVDSCommand, return: MigratingFrom, log id: 4282c86d >2012-09-02 18:29:13,289 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-5) [1b411059] vds::refreshVmList vm id 5449d6cf-0e31-44b5-b0a9-65036f122662 is migrating to vds camel-vdsb.qa.lab.tlv.redhat.com ignoring it in the refresh till migration is done >2012-09-02 18:29:15,413 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-4) [20d6286c] vds::refreshVmList vm id 5449d6cf-0e31-44b5-b0a9-65036f122662 is migrating to vds camel-vdsb.qa.lab.tlv.redhat.com ignoring it in the refresh till migration is done >2012-09-02 18:29:16,780 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (QuartzScheduler_Worker-64) [7f1d7123] START, DestroyVDSCommand(vdsId = b32eacf0-f4e9-11e1-973c-001a4a169705, vmId=5449d6cf-0e31-44b5-b0a9-65036f122662, force=false, secondsToWait=0, gracefully=false), log id: 451acb0c >2012-09-02 18:29:16,848 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (QuartzScheduler_Worker-64) [7f1d7123] FINISH, DestroyVDSCommand, log id: 451acb0c >2012-09-02 18:29:16,849 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-64) [7f1d7123] Setting VM testVm 5449d6cf-0e31-44b5-b0a9-65036f122662 to status unknown >2012-09-02 18:29:16,849 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-64) [7f1d7123] vm testVm running in db and not running in vds - add to rerun treatment. vds green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 18:29:16,908 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (QuartzScheduler_Worker-1) [15ef6589] START, FullListVdsCommand(vdsId = 3880c572-f4f0-11e1-9a52-001a4a169705, vds=null, vmIds=[5449d6cf-0e31-44b5-b0a9-65036f122662]), log id: 546dbb3b >2012-09-02 18:29:16,927 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (QuartzScheduler_Worker-1) [15ef6589] FINISH, FullListVdsCommand, return: [Lorg.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcStruct;@701f49ca, log id: 546dbb3b >2012-09-02 18:29:22,085 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-9) [5d8d8d2c] starting ProcessDomainRecovery for domain 56a64ce1-90ab-4576-86fa-c269160aaa38 >2012-09-02 18:29:22,089 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-9) [5d8d8d2c] vds green-vdsa.qa.lab.tlv.redhat.com reported domain 56a64ce1-90ab-4576-86fa-c269160aaa38:ISCSIDataDomain as in problem, vds is in status NonOperational, no need to move to nonoperational >2012-09-02 18:29:23,015 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [1cdd254a] domain 56a64ce1-90ab-4576-86fa-c269160aaa38 in problem. vds: green-vdsa.qa.lab.tlv.redhat.com >2012-09-02 18:29:32,294 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-33) [405ae04a] starting ProcessDomainRecovery for domain 56a64ce1-90ab-4576-86fa-c269160aaa38 >2012-09-02 18:29:32,298 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-33) [405ae04a] vds green-vdsa.qa.lab.tlv.redhat.com reported domain 56a64ce1-90ab-4576-86fa-c269160aaa38:ISCSIDataDomain as in problem, vds is in status NonOperational, no need to move to nonoperational >2012-09-02 18:30:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-91) [70f89c76] Autorecovering hosts is disabled, skipping >2012-09-02 18:30:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-91) [70f89c76] Autorecovering storage domains is disabled, skipping >2012-09-02 18:35:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-20) [35fd83fa] Autorecovering hosts is disabled, skipping >2012-09-02 18:35:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-20) [35fd83fa] Autorecovering storage domains is disabled, skipping >2012-09-02 18:40:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-26) [c2cfb36] Autorecovering hosts is disabled, skipping >2012-09-02 18:40:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-26) [c2cfb36] Autorecovering storage domains is disabled, skipping
<b>You cannot view the attachment while viewing its details because your browser does not support IFRAMEs. <a href="attachment.cgi?id=609123">View the attachment on a separate page</a>.</b>
View Attachment As Raw
Actions:
View
Attachments on
bug 853747
: 609123