Login
[x]
Log in using an account from:
Fedora Account System
Red Hat Associate
Red Hat Customer
Or login using a Red Hat Bugzilla account
Forgot Password
Login:
Hide Forgot
Create an Account
Red Hat Bugzilla – Attachment 640889 Details for
Bug 874660
Hosts from another data center displayed when clicked on "Hosts" in tree view for cluster names with a single difference.
[?]
New
Simple Search
Advanced Search
My Links
Browse
Requests
Reports
Current State
Search
Tabular reports
Graphical reports
Duplicates
Other Reports
User Changes
Plotly Reports
Bug Status
Bug Severity
Non-Defaults
|
Product Dashboard
Help
Page Help!
Bug Writing Guidelines
What's new
Browser Support Policy
5.0.4.rh83 Release notes
FAQ
Guides index
User guide
Web Services
Contact
Legal
This site requires JavaScript to be enabled to function correctly, please enable it.
Engine log
engine.log (text/plain), 2.97 MB, created by
Gowrishankar Rajaiyan
on 2012-11-08 15:39:33 UTC
(
hide
)
Description:
Engine log
Filename:
MIME Type:
Creator:
Gowrishankar Rajaiyan
Created:
2012-11-08 15:39:33 UTC
Size:
2.97 MB
patch
obsolete
>2012-11-08 02:12:08,256 ERROR [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-3) Error in getting DB connection. The database is inaccessible. Original exception is: DataAccessResourceFailureException: Error retreiving database metadata; nested exception is org.springframework.jdbc.support.MetaDataAccessException: Could not get Connection for extracting meta data; nested exception is org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:/ENGINEDataSource >2012-11-08 02:12:13,266 ERROR [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-3) Error in getting DB connection. The database is inaccessible. Original exception is: DataAccessResourceFailureException: Error retreiving database metadata; nested exception is org.springframework.jdbc.support.MetaDataAccessException: Could not get Connection for extracting meta data; nested exception is org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:/ENGINEDataSource >2012-11-08 02:12:18,274 ERROR [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-3) Error in getting DB connection. The database is inaccessible. Original exception is: DataAccessResourceFailureException: Error retreiving database metadata; nested exception is org.springframework.jdbc.support.MetaDataAccessException: Could not get Connection for extracting meta data; nested exception is org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:/ENGINEDataSource >2012-11-08 02:12:23,288 ERROR [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-3) Error in getting DB connection. The database is inaccessible. Original exception is: DataAccessResourceFailureException: Error retreiving database metadata; nested exception is org.springframework.jdbc.support.MetaDataAccessException: Could not get Connection for extracting meta data; nested exception is org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:/ENGINEDataSource >2012-11-08 02:12:28,301 ERROR [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-3) Error in getting DB connection. The database is inaccessible. Original exception is: DataAccessResourceFailureException: Error retreiving database metadata; nested exception is org.springframework.jdbc.support.MetaDataAccessException: Could not get Connection for extracting meta data; nested exception is org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:/ENGINEDataSource >2012-11-08 02:12:33,318 ERROR [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-3) Error in getting DB connection. The database is inaccessible. Original exception is: DataAccessResourceFailureException: Error retreiving database metadata; nested exception is org.springframework.jdbc.support.MetaDataAccessException: Could not get Connection for extracting meta data; nested exception is org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:/ENGINEDataSource >2012-11-08 02:12:38,332 ERROR [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-3) Error in getting DB connection. The database is inaccessible. Original exception is: DataAccessResourceFailureException: Error retreiving database metadata; nested exception is org.springframework.jdbc.support.MetaDataAccessException: Could not get Connection for extracting meta data; nested exception is org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:/ENGINEDataSource >2012-11-08 02:12:43,344 ERROR [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-3) Error in getting DB connection. The database is inaccessible. Original exception is: DataAccessResourceFailureException: Error retreiving database metadata; nested exception is org.springframework.jdbc.support.MetaDataAccessException: Could not get Connection for extracting meta data; nested exception is org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:/ENGINEDataSource >2012-11-08 02:12:48,353 ERROR [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-3) Error in getting DB connection. The database is inaccessible. Original exception is: DataAccessResourceFailureException: Error retreiving database metadata; nested exception is org.springframework.jdbc.support.MetaDataAccessException: Could not get Connection for extracting meta data; nested exception is org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:/ENGINEDataSource >2012-11-08 02:12:53,362 ERROR [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-3) Error in getting DB connection. The database is inaccessible. Original exception is: DataAccessResourceFailureException: Error retreiving database metadata; nested exception is org.springframework.jdbc.support.MetaDataAccessException: Could not get Connection for extracting meta data; nested exception is org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:/ENGINEDataSource >2012-11-08 02:12:58,368 ERROR [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-3) Error in getting DB connection. The database is inaccessible. Original exception is: DataAccessResourceFailureException: Error retreiving database metadata; nested exception is org.springframework.jdbc.support.MetaDataAccessException: Could not get Connection for extracting meta data; nested exception is org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:/ENGINEDataSource >2012-11-08 02:13:03,376 ERROR [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-3) Error in getting DB connection. The database is inaccessible. Original exception is: DataAccessResourceFailureException: Error retreiving database metadata; nested exception is org.springframework.jdbc.support.MetaDataAccessException: Could not get Connection for extracting meta data; nested exception is org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:/ENGINEDataSource >2012-11-08 02:13:08,388 ERROR [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-3) Error in getting DB connection. The database is inaccessible. Original exception is: DataAccessResourceFailureException: Error retreiving database metadata; nested exception is org.springframework.jdbc.support.MetaDataAccessException: Could not get Connection for extracting meta data; nested exception is org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:/ENGINEDataSource >2012-11-08 02:13:13,406 ERROR [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-3) Error in getting DB connection. The database is inaccessible. Original exception is: DataAccessResourceFailureException: Error retreiving database metadata; nested exception is org.springframework.jdbc.support.MetaDataAccessException: Could not get Connection for extracting meta data; nested exception is org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:/ENGINEDataSource >2012-11-08 02:13:18,412 ERROR [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-3) Error in getting DB connection. The database is inaccessible. Original exception is: DataAccessResourceFailureException: Error retreiving database metadata; nested exception is org.springframework.jdbc.support.MetaDataAccessException: Could not get Connection for extracting meta data; nested exception is org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:/ENGINEDataSource >2012-11-08 02:13:23,420 ERROR [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-3) Error in getting DB connection. The database is inaccessible. Original exception is: DataAccessResourceFailureException: Error retreiving database metadata; nested exception is org.springframework.jdbc.support.MetaDataAccessException: Could not get Connection for extracting meta data; nested exception is org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:/ENGINEDataSource >2012-11-08 02:13:28,428 ERROR [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-3) Error in getting DB connection. The database is inaccessible. Original exception is: DataAccessResourceFailureException: Error retreiving database metadata; nested exception is org.springframework.jdbc.support.MetaDataAccessException: Could not get Connection for extracting meta data; nested exception is org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:/ENGINEDataSource >2012-11-08 02:13:33,437 ERROR [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-3) Error in getting DB connection. The database is inaccessible. Original exception is: DataAccessResourceFailureException: Error retreiving database metadata; nested exception is org.springframework.jdbc.support.MetaDataAccessException: Could not get Connection for extracting meta data; nested exception is org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:/ENGINEDataSource >2012-11-08 02:13:38,451 ERROR [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-3) Error in getting DB connection. The database is inaccessible. Original exception is: DataAccessResourceFailureException: Error retreiving database metadata; nested exception is org.springframework.jdbc.support.MetaDataAccessException: Could not get Connection for extracting meta data; nested exception is org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:/ENGINEDataSource >2012-11-08 02:13:43,463 ERROR [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-3) Error in getting DB connection. The database is inaccessible. Original exception is: DataAccessResourceFailureException: Error retreiving database metadata; nested exception is org.springframework.jdbc.support.MetaDataAccessException: Could not get Connection for extracting meta data; nested exception is org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:/ENGINEDataSource >2012-11-08 02:13:48,473 ERROR [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-3) Error in getting DB connection. The database is inaccessible. Original exception is: DataAccessResourceFailureException: Error retreiving database metadata; nested exception is org.springframework.jdbc.support.MetaDataAccessException: Could not get Connection for extracting meta data; nested exception is org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:/ENGINEDataSource >2012-11-08 02:13:53,480 ERROR [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-3) Error in getting DB connection. The database is inaccessible. Original exception is: DataAccessResourceFailureException: Error retreiving database metadata; nested exception is org.springframework.jdbc.support.MetaDataAccessException: Could not get Connection for extracting meta data; nested exception is org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:/ENGINEDataSource >2012-11-08 02:13:58,492 ERROR [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-3) Error in getting DB connection. The database is inaccessible. Original exception is: DataAccessResourceFailureException: Error retreiving database metadata; nested exception is org.springframework.jdbc.support.MetaDataAccessException: Could not get Connection for extracting meta data; nested exception is org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:/ENGINEDataSource >2012-11-08 02:14:03,499 ERROR [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-3) Error in getting DB connection. The database is inaccessible. Original exception is: DataAccessResourceFailureException: Error retreiving database metadata; nested exception is org.springframework.jdbc.support.MetaDataAccessException: Could not get Connection for extracting meta data; nested exception is org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:/ENGINEDataSource >2012-11-08 02:14:08,507 ERROR [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-3) Error in getting DB connection. The database is inaccessible. Original exception is: DataAccessResourceFailureException: Error retreiving database metadata; nested exception is org.springframework.jdbc.support.MetaDataAccessException: Could not get Connection for extracting meta data; nested exception is org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:/ENGINEDataSource >2012-11-08 02:14:13,517 ERROR [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-3) Error in getting DB connection. The database is inaccessible. Original exception is: DataAccessResourceFailureException: Error retreiving database metadata; nested exception is org.springframework.jdbc.support.MetaDataAccessException: Could not get Connection for extracting meta data; nested exception is org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:/ENGINEDataSource >2012-11-08 02:14:18,527 ERROR [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-3) Error in getting DB connection. The database is inaccessible. Original exception is: DataAccessResourceFailureException: Error retreiving database metadata; nested exception is org.springframework.jdbc.support.MetaDataAccessException: Could not get Connection for extracting meta data; nested exception is org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:/ENGINEDataSource >2012-11-08 02:14:23,536 ERROR [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-3) Error in getting DB connection. The database is inaccessible. Original exception is: DataAccessResourceFailureException: Error retreiving database metadata; nested exception is org.springframework.jdbc.support.MetaDataAccessException: Could not get Connection for extracting meta data; nested exception is org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:/ENGINEDataSource >2012-11-08 02:14:28,546 ERROR [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-3) Error in getting DB connection. The database is inaccessible. Original exception is: DataAccessResourceFailureException: Error retreiving database metadata; nested exception is org.springframework.jdbc.support.MetaDataAccessException: Could not get Connection for extracting meta data; nested exception is org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:/ENGINEDataSource >2012-11-08 02:14:33,555 ERROR [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-3) Error in getting DB connection. The database is inaccessible. Original exception is: DataAccessResourceFailureException: Error retreiving database metadata; nested exception is org.springframework.jdbc.support.MetaDataAccessException: Could not get Connection for extracting meta data; nested exception is org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:/ENGINEDataSource >2012-11-08 02:14:38,563 ERROR [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-3) Error in getting DB connection. The database is inaccessible. Original exception is: DataAccessResourceFailureException: Error retreiving database metadata; nested exception is org.springframework.jdbc.support.MetaDataAccessException: Could not get Connection for extracting meta data; nested exception is org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:/ENGINEDataSource >2012-11-08 02:14:43,572 ERROR [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-3) Error in getting DB connection. The database is inaccessible. Original exception is: DataAccessResourceFailureException: Error retreiving database metadata; nested exception is org.springframework.jdbc.support.MetaDataAccessException: Could not get Connection for extracting meta data; nested exception is org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:/ENGINEDataSource >2012-11-08 02:14:48,580 ERROR [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-3) Error in getting DB connection. The database is inaccessible. Original exception is: DataAccessResourceFailureException: Error retreiving database metadata; nested exception is org.springframework.jdbc.support.MetaDataAccessException: Could not get Connection for extracting meta data; nested exception is org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:/ENGINEDataSource >2012-11-08 02:14:53,592 ERROR [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-3) Error in getting DB connection. The database is inaccessible. Original exception is: DataAccessResourceFailureException: Error retreiving database metadata; nested exception is org.springframework.jdbc.support.MetaDataAccessException: Could not get Connection for extracting meta data; nested exception is org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:/ENGINEDataSource >2012-11-08 02:14:58,604 ERROR [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-3) Error in getting DB connection. The database is inaccessible. Original exception is: DataAccessResourceFailureException: Error retreiving database metadata; nested exception is org.springframework.jdbc.support.MetaDataAccessException: Could not get Connection for extracting meta data; nested exception is org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:/ENGINEDataSource >2012-11-08 02:15:03,621 ERROR [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-3) Error in getting DB connection. The database is inaccessible. Original exception is: DataAccessResourceFailureException: Error retreiving database metadata; nested exception is org.springframework.jdbc.support.MetaDataAccessException: Could not get Connection for extracting meta data; nested exception is org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:/ENGINEDataSource >2012-11-08 02:15:08,634 ERROR [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-3) Error in getting DB connection. The database is inaccessible. Original exception is: DataAccessResourceFailureException: Error retreiving database metadata; nested exception is org.springframework.jdbc.support.MetaDataAccessException: Could not get Connection for extracting meta data; nested exception is org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:/ENGINEDataSource >2012-11-08 02:15:13,653 ERROR [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-3) Error in getting DB connection. The database is inaccessible. Original exception is: DataAccessResourceFailureException: Error retreiving database metadata; nested exception is org.springframework.jdbc.support.MetaDataAccessException: Could not get Connection for extracting meta data; nested exception is org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:/ENGINEDataSource >2012-11-08 02:15:18,666 ERROR [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-3) Error in getting DB connection. The database is inaccessible. Original exception is: DataAccessResourceFailureException: Error retreiving database metadata; nested exception is org.springframework.jdbc.support.MetaDataAccessException: Could not get Connection for extracting meta data; nested exception is org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:/ENGINEDataSource >2012-11-08 02:19:54,825 INFO [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-1) Start time: 11/8/12 2:19 AM >2012-11-08 02:19:54,915 WARN [org.ovirt.engine.core.utils.ConfigUtilsBase] (MSC service thread 1-1) Could not find enum value for option: CbcCheckOnVdsChange >2012-11-08 02:19:54,939 WARN [org.ovirt.engine.core.utils.ConfigUtilsBase] (MSC service thread 1-1) Could not find enum value for option: CAEngineKey >2012-11-08 02:19:54,939 WARN [org.ovirt.engine.core.utils.ConfigUtilsBase] (MSC service thread 1-1) Could not find enum value for option: ENGINEEARLib >2012-11-08 02:19:54,954 WARN [org.ovirt.engine.core.utils.ConfigUtilsBase] (MSC service thread 1-1) Could not find enum value for option: MinimalETLVersion >2012-11-08 02:19:55,061 WARN [org.ovirt.engine.core.utils.ConfigUtilsBase] (MSC service thread 1-1) Could not find enum value for option: ScriptsPath >2012-11-08 02:19:55,062 WARN [org.ovirt.engine.core.utils.ConfigUtilsBase] (MSC service thread 1-1) Could not find enum value for option: SQLServerI18NPrefix >2012-11-08 02:19:55,545 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (MSC service thread 1-1) Failed to decrypt Data must start with zero >2012-11-08 02:19:55,545 ERROR [org.ovirt.engine.core.dal.dbbroker.generic.DBConfigUtils] (MSC service thread 1-1) Failed to decrypt value for property TruststorePass will be used encrypted value >2012-11-08 02:19:55,615 INFO [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-1) VDSBrokerFrontend: 11/8/12 2:19 AM >2012-11-08 02:19:55,628 INFO [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-1) CpuFlagsManager: 11/8/12 2:19 AM >2012-11-08 02:19:55,629 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (MSC service thread 1-1) Setting audit clean up manager to run at: 35 35 3 * * ? >2012-11-08 02:19:55,654 INFO [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-1) AuditLogCleanupManager: 11/8/12 2:19 AM >2012-11-08 02:19:55,657 INFO [org.ovirt.engine.core.bll.TagsDirector] (MSC service thread 1-1) TagsDirector initialization >2012-11-08 02:19:55,687 INFO [org.ovirt.engine.core.bll.TagsDirector] (MSC service thread 1-1) Tag root added to tree >2012-11-08 02:19:55,705 INFO [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-1) TagsDirector: 11/8/12 2:19 AM >2012-11-08 02:19:55,716 INFO [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-1) IsoDomainListSyncronizer: 11/8/12 2:19 AM >2012-11-08 02:19:55,744 INFO [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-1) InitHandlers: 11/8/12 2:19 AM >2012-11-08 02:19:55,766 WARN [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (MSC service thread 1-1) Code VDS_FENCING_OPERATION_FAILED appears more then once in string table. >2012-11-08 02:19:55,767 INFO [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-1) ErrorTranslator: 11/8/12 2:19 AM >2012-11-08 02:19:55,769 INFO [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-1) VdsErrorTranslator: 11/8/12 2:19 AM >2012-11-08 02:19:55,770 INFO [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-1) Mark uncompleted jobs as UNKNOWN: 11/8/12 2:19 AM >2012-11-08 02:19:55,813 INFO [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-1) JobRepositoryCleanupManager: 11/8/12 2:19 AM >2012-11-08 02:19:55,815 INFO [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-1) AutoRecoveryManager: 11/8/12 2:19 AM >2012-11-08 02:19:55,821 INFO [org.ovirt.engine.core.bll.Backend] (MSC service thread 1-1) ExecutionMessageDirector: 11/8/12 2:19 AM >2012-11-08 02:19:55,847 INFO [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (MSC service thread 1-2) InitResourceManager: 11/8/12 2:19 AM >2012-11-08 02:19:55,848 INFO [org.ovirt.engine.core.bll.adbroker.UsersDomainsCacheManagerService] (MSC service thread 1-4) UsersDomainsCacheManager: 11/8/12 2:19 AM >2012-11-08 02:19:55,850 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (MSC service thread 1-2) ResourceManager::ResourceManager::Entered >2012-11-08 02:19:55,890 INFO [org.ovirt.engine.core.bll.adbroker.UsersDomainsCacheManagerService] (MSC service thread 1-4) DbUserCacheManager: 11/8/12 2:19 AM >2012-11-08 02:19:55,919 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-1) DbUserCacheManager::refreshAllUserData() - entered >2012-11-08 02:19:56,001 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (MSC service thread 1-2) AsyncTaskManager: Initialization of AsyncTaskManager completed successfully. >2012-11-08 02:19:56,002 INFO [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (MSC service thread 1-2) AsyncTaskManager: 11/8/12 2:19 AM >2012-11-08 02:19:56,003 INFO [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (MSC service thread 1-2) VdsLoadBalancer: 11/8/12 2:19 AM >2012-11-08 02:19:56,020 INFO [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (pool-4-thread-1) MacPoolManager started: 11/8/12 2:19 AM >2012-11-08 02:19:56,144 INFO [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (pool-4-thread-1) MacPoolManager finished: 11/8/12 2:19 AM >2012-11-08 02:20:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-2) Autorecovering hosts is disabled, skipping >2012-11-08 02:20:00,007 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-2) Autorecovering storage domains is disabled, skipping >2012-11-08 02:25:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-80) Autorecovering hosts is disabled, skipping >2012-11-08 02:25:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-80) Autorecovering storage domains is disabled, skipping >2012-11-08 02:29:18,394 INFO [org.ovirt.engine.core.bll.LoginAdminUserCommand] (ajp-/127.0.0.1:8702-1) Checking if user admin@internal is an admin, result true >2012-11-08 02:29:18,396 INFO [org.ovirt.engine.core.bll.LoginAdminUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginAdminUserCommand internal: false. >2012-11-08 02:29:18,436 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-1) AuditLogType: UNASSIGNED not exist in string table >2012-11-08 02:29:18,437 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-1) AuditLogType: VDS_NETWORK_MTU_DIFFER_FROM_LOGICAL_NETWORK not exist in string table >2012-11-08 02:29:18,437 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-1) AuditLogType: STORAGE_ACTIVATE_ASYNC not exist in string table >2012-11-08 02:29:18,437 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-1) AuditLogType: DWH_STOPPED not exist in string table >2012-11-08 02:29:18,437 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-1) AuditLogType: DWH_STARTED not exist in string table >2012-11-08 02:29:18,438 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-1) AuditLogType: DWH_ERROR not exist in string table >2012-11-08 02:29:18,441 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-1) AuditLogType: USER_ATTACH_DISK_TO_VM not have severity. Assumed Normal >2012-11-08 02:29:18,441 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-1) AuditLogType: USER_FAILED_ATTACH_DISK_TO_VM not have severity. Assumed Normal >2012-11-08 02:29:18,441 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-1) AuditLogType: USER_DETACH_DISK_FROM_VM not have severity. Assumed Normal >2012-11-08 02:29:18,441 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-1) AuditLogType: USER_FAILED_DETACH_DISK_FROM_VM not have severity. Assumed Normal >2012-11-08 02:29:18,441 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-1) AuditLogType: GLUSTER_HOST_ADD_FAILED not have severity. Assumed Normal >2012-11-08 02:29:18,441 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-1) AuditLogType: VM_MIGRATION_START_SYSTEM_INITIATED not have severity. Assumed Normal >2012-11-08 02:29:18,442 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-1) AuditLogType: VDS_NETWORK_MTU_DIFFER_FROM_LOGICAL_NETWORK not have severity. Assumed Normal >2012-11-08 02:29:41,671 WARN [org.ovirt.engine.core.dal.dbbroker.generic.DBConfigUtils] (ajp-/127.0.0.1:8702-8) Adding new value to configuration cache. >2012-11-08 02:29:41,671 WARN [org.ovirt.engine.core.dal.dbbroker.generic.DBConfigUtils] (ajp-/127.0.0.1:8702-8) Didnt find the value of DBPagingType in DB for version general - using default: Offset >2012-11-08 02:29:41,672 WARN [org.ovirt.engine.core.dal.dbbroker.generic.DBConfigUtils] (ajp-/127.0.0.1:8702-8) Adding new value to configuration cache. >2012-11-08 02:29:41,672 WARN [org.ovirt.engine.core.dal.dbbroker.generic.DBConfigUtils] (ajp-/127.0.0.1:8702-8) Didnt find the value of DBPagingSyntax in DB for version general - using default: OFFSET (%1$s -1) LIMIT %2$s >2012-11-08 02:29:41,674 WARN [org.ovirt.engine.core.dal.dbbroker.generic.DBConfigUtils] (ajp-/127.0.0.1:8702-8) Adding new value to configuration cache. >2012-11-08 02:29:41,674 WARN [org.ovirt.engine.core.dal.dbbroker.generic.DBConfigUtils] (ajp-/127.0.0.1:8702-8) Didnt find the value of DBSearchTemplate in DB for version general - using default: SELECT * FROM (%2$s) %1$s) as T1 %3$s >2012-11-08 02:30:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-58) Autorecovering hosts is disabled, skipping >2012-11-08 02:30:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-58) Autorecovering storage domains is disabled, skipping >2012-11-08 02:31:46,790 INFO [org.ovirt.engine.core.bll.storage.AddEmptyStoragePoolCommand] (ajp-/127.0.0.1:8702-2) [47fad7af] Running command: AddEmptyStoragePoolCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2012-11-08 02:32:18,467 INFO [org.ovirt.engine.core.bll.AddVdsGroupCommand] (ajp-/127.0.0.1:8702-12) [4228abeb] Running command: AddVdsGroupCommand internal: false. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 02:33:14,603 INFO [org.ovirt.engine.core.bll.storage.AddEmptyStoragePoolCommand] (ajp-/127.0.0.1:8702-3) [3b22c3ca] Running command: AddEmptyStoragePoolCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2012-11-08 02:33:50,838 INFO [org.ovirt.engine.core.bll.storage.UpdateStoragePoolCommand] (ajp-/127.0.0.1:8702-8) [3b64ba65] Running command: UpdateStoragePoolCommand internal: false. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 02:33:59,890 WARN [org.ovirt.engine.core.dal.dbbroker.generic.DBConfigUtils] (ajp-/127.0.0.1:8702-6) Adding new value to configuration cache. >2012-11-08 02:33:59,891 WARN [org.ovirt.engine.core.dal.dbbroker.generic.DBConfigUtils] (ajp-/127.0.0.1:8702-6) Didnt find the value of DBI18NPrefix in DB for version general - using default: >2012-11-08 02:34:24,921 WARN [org.ovirt.engine.core.bll.AddVdsGroupCommand] (ajp-/127.0.0.1:8702-9) [23470b8a] CanDoAction of action AddVdsGroup failed. Reasons:VAR__TYPE__CLUSTER,VAR__ACTION__CREATE,VDS_GROUP_CANNOT_DO_ACTION_NAME_IN_USE >2012-11-08 02:34:38,335 INFO [org.ovirt.engine.core.bll.AddVdsGroupCommand] (ajp-/127.0.0.1:8702-7) [3da861de] Running command: AddVdsGroupCommand internal: false. Entities affected : ID: 1eaedbbb-1c0e-4028-9b0e-adac7e07d25f Type: StoragePool >2012-11-08 02:35:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-35) Autorecovering hosts is disabled, skipping >2012-11-08 02:35:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-35) Autorecovering storage domains is disabled, skipping >2012-11-08 02:35:52,579 INFO [org.ovirt.engine.core.bll.storage.AddEmptyStoragePoolCommand] (ajp-/127.0.0.1:8702-5) [6f7fc2d] Running command: AddEmptyStoragePoolCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2012-11-08 02:36:15,079 INFO [org.ovirt.engine.core.bll.AddVdsGroupCommand] (ajp-/127.0.0.1:8702-10) [6647e086] Running command: AddVdsGroupCommand internal: false. Entities affected : ID: f4c98a0b-0578-485f-90f4-743df3d05d7e Type: StoragePool >2012-11-08 02:36:49,978 WARN [org.ovirt.engine.core.bll.AddVdsGroupCommand] (ajp-/127.0.0.1:8702-11) [2acddbca] CanDoAction of action AddVdsGroup failed. Reasons:VAR__TYPE__CLUSTER,VAR__ACTION__CREATE,VDS_GROUP_ENABLING_BOTH_VIRT_AND_GLUSTER_SERVICES_NOT_ALLOWED >2012-11-08 02:37:27,897 INFO [org.ovirt.engine.core.bll.AddVdsGroupCommand] (ajp-/127.0.0.1:8702-4) [71691855] Running command: AddVdsGroupCommand internal: false. Entities affected : ID: f4c98a0b-0578-485f-90f4-743df3d05d7e Type: StoragePool >2012-11-08 02:38:48,684 ERROR [org.ovirt.engine.core.bll.FencingExecutor] (ajp-/127.0.0.1:8702-8) Failed to run Power Management command on Host , no running proxy Host was found. >2012-11-08 02:38:56,668 ERROR [org.ovirt.engine.core.bll.FencingExecutor] (ajp-/127.0.0.1:8702-1) Failed to run Power Management command on Host , no running proxy Host was found. >2012-11-08 02:39:00,907 ERROR [org.ovirt.engine.core.bll.FencingExecutor] (ajp-/127.0.0.1:8702-10) Failed to run Power Management command on Host , no running proxy Host was found. >2012-11-08 02:39:33,138 ERROR [org.ovirt.engine.core.bll.FencingExecutor] (ajp-/127.0.0.1:8702-7) Failed to run Power Management command on Host , no running proxy Host was found. >2012-11-08 02:40:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-13) Autorecovering hosts is disabled, skipping >2012-11-08 02:40:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-13) Autorecovering storage domains is disabled, skipping >2012-11-08 02:40:17,816 ERROR [org.ovirt.engine.core.bll.FencingExecutor] (ajp-/127.0.0.1:8702-12) Failed to run Power Management command on Host , no running proxy Host was found. >2012-11-08 02:40:20,176 ERROR [org.ovirt.engine.core.bll.FencingExecutor] (ajp-/127.0.0.1:8702-9) Failed to run Power Management command on Host , no running proxy Host was found. >2012-11-08 02:40:27,948 WARN [org.ovirt.engine.core.dal.dbbroker.generic.DBConfigUtils] (ajp-/127.0.0.1:8702-10) [7de454a] Adding new value to configuration cache. >2012-11-08 02:40:27,948 WARN [org.ovirt.engine.core.dal.dbbroker.generic.DBConfigUtils] (ajp-/127.0.0.1:8702-10) [7de454a] Didnt find the value of BootstrapNodeIDCommand in DB for version general - using default: IDFILE=/etc/vdsm/vdsm.id; if [ -r "${IDFILE}" ]; then cat "${IDFILE}"; else UUID="$(dmidecode -s system-uuid 2> /dev/null | sed -e 's/.*Not.*//' )"; if [ -z "${UUID}" ]; then UUID="$(uuidgen 2> /dev/null)" && mkdir -p "$(dirname "${IDFILE}")" && echo "${UUID}" > "${IDFILE}" && chmod 0644 "${IDFILE}"; fi; [ -n "${UUID}" ] && echo "${UUID}"; fi >2012-11-08 02:40:27,949 INFO [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (ajp-/127.0.0.1:8702-10) SSH execute 10.70.36.35 'IDFILE=/etc/vdsm/vdsm.id; if [ -r "${IDFILE}" ]; then cat "${IDFILE}"; else UUID="$(dmidecode -s system-uuid 2> /dev/null | sed -e 's/.*Not.*//' )"; if [ -z "${UUID}" ]; then UUID="$(uuidgen 2> /dev/null)" && mkdir -p "$(dirname "${IDFILE}")" && echo "${UUID}" > "${IDFILE}" && chmod 0644 "${IDFILE}"; fi; [ -n "${UUID}" ] && echo "${UUID}"; fi' >2012-11-08 02:40:28,014 INFO [org.ovirt.engine.core.bll.AddVdsCommand] (ajp-/127.0.0.1:8702-10) [7de454a] Running command: AddVdsCommand internal: false. Entities affected : ID: 2559657e-2977-11e2-a832-52540035ed48 Type: VdsGroups >2012-11-08 02:40:28,093 INFO [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp-/127.0.0.1:8702-10) [1b738151] Running command: AddVdsSpmIdCommand internal: true. Entities affected : ID: 90b581c2-2977-11e2-82cd-52540035ed48 Type: VDS >2012-11-08 02:40:28,100 INFO [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp-/127.0.0.1:8702-10) [1b738151] Lock freed to object EngineLock [exclusiveLocks= key: f4c98a0b-0578-485f-90f4-743df3d05d7e value: REGISTER_VDS >, sharedLocks= ] >2012-11-08 02:40:28,103 ERROR [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp-/127.0.0.1:8702-10) [1b738151] Cannot get vdsManager for vdsid=90b581c2-2977-11e2-82cd-52540035ed48 >2012-11-08 02:40:28,115 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (ajp-/127.0.0.1:8702-10) [1b738151] START, RemoveVdsVDSCommand(HostName = rhs-client11.lab.eng.blr.redhat.com, HostId = 90b581c2-2977-11e2-82cd-52540035ed48), log id: 685a882e >2012-11-08 02:40:28,116 ERROR [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp-/127.0.0.1:8702-10) [1b738151] Cannot get vdsManager for vdsid=90b581c2-2977-11e2-82cd-52540035ed48 >2012-11-08 02:40:28,116 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (ajp-/127.0.0.1:8702-10) [1b738151] FINISH, RemoveVdsVDSCommand, log id: 685a882e >2012-11-08 02:40:28,116 ERROR [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp-/127.0.0.1:8702-10) [1b738151] Cannot get vdsManager for vdsid=90b581c2-2977-11e2-82cd-52540035ed48 >2012-11-08 02:40:28,127 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-10) [1b738151] START, AddVdsVDSCommand(HostName = rhs-client11.lab.eng.blr.redhat.com, HostId = 90b581c2-2977-11e2-82cd-52540035ed48), log id: 32c835b3 >2012-11-08 02:40:28,127 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-10) [1b738151] AddVds - entered , starting logic to add VDS 90b581c2-2977-11e2-82cd-52540035ed48 >2012-11-08 02:40:28,148 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-10) [1b738151] AddVds - VDS 90b581c2-2977-11e2-82cd-52540035ed48 was added, will try to add it to the resource manager >2012-11-08 02:40:28,152 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ajp-/127.0.0.1:8702-10) [1b738151] Eneterd VdsManager:constructor >2012-11-08 02:40:28,152 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ajp-/127.0.0.1:8702-10) [1b738151] vdsBroker(10.70.36.35,54,321) >2012-11-08 02:40:28,235 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp-/127.0.0.1:8702-10) [1b738151] ResourceManager::AddVds - VDS 90b581c2-2977-11e2-82cd-52540035ed48 was added to the Resource Manager >2012-11-08 02:40:28,236 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-10) [1b738151] FINISH, AddVdsVDSCommand, log id: 32c835b3 >2012-11-08 02:40:28,269 ERROR [org.ovirt.engine.core.bll.FencingExecutor] (ajp-/127.0.0.1:8702-10) [1b738151] Failed to run Power Management command on Host rhs-client11.lab.eng.blr.redhat.com, no running proxy Host was found. >2012-11-08 02:40:28,324 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-4-thread-3) [3a60121b] Running command: InstallVdsCommand internal: true. Entities affected : ID: 90b581c2-2977-11e2-82cd-52540035ed48 Type: VDS >2012-11-08 02:40:28,327 WARN [org.ovirt.engine.core.dal.dbbroker.generic.DBConfigUtils] (pool-4-thread-3) [3a60121b] Adding new value to configuration cache. >2012-11-08 02:40:28,327 WARN [org.ovirt.engine.core.dal.dbbroker.generic.DBConfigUtils] (pool-4-thread-3) [3a60121b] Didnt find the value of BootstrapCommand in DB for version general - using default: umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar -C "${MYTMP}" --no-same-permissions -o -x && "${MYTMP}"/setup -c 'ssl={server_SSL_enabled};management_port={management_port}' -O '{OrganizationName}' -t {utc_time} {OverrideFirewall} -S {SSHKey} {EnginePort} -b {virt-placeholder} -B {BridgeName} {gluster-placeholder} {URL1} {URL1} {vds-server} {GUID} {RunFlag} >2012-11-08 02:40:28,328 WARN [org.ovirt.engine.core.dal.dbbroker.generic.DBConfigUtils] (pool-4-thread-3) [3a60121b] Adding new value to configuration cache. >2012-11-08 02:40:28,328 WARN [org.ovirt.engine.core.dal.dbbroker.generic.DBConfigUtils] (pool-4-thread-3) [3a60121b] Didnt find the value of BootstrapPackageName in DB for version general - using default: vdsm-bootstrap-2.tar >2012-11-08 02:40:28,329 WARN [org.ovirt.engine.core.dal.dbbroker.generic.DBConfigUtils] (pool-4-thread-3) [3a60121b] Adding new value to configuration cache. >2012-11-08 02:40:28,329 WARN [org.ovirt.engine.core.dal.dbbroker.generic.DBConfigUtils] (pool-4-thread-3) [3a60121b] Didnt find the value of BootstrapPackageDirectory in DB for version general - using default: /usr/share/vdsm-bootstrap/interface-2 >2012-11-08 02:40:28,329 WARN [org.ovirt.engine.core.dal.dbbroker.generic.DBConfigUtils] (pool-4-thread-3) [3a60121b] Adding new value to configuration cache. >2012-11-08 02:40:28,329 WARN [org.ovirt.engine.core.dal.dbbroker.generic.DBConfigUtils] (pool-4-thread-3) [3a60121b] Didnt find the value of BootstrapCacheRefreshInterval in DB for version general - using default: 10,000 >2012-11-08 02:40:28,332 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-4-thread-3) [3a60121b] Before Installation pool-4-thread-3 >2012-11-08 02:40:28,332 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-3) [3a60121b] Installation of 10.70.36.35. Executing installation stage. (Stage: Starting Host installation) >2012-11-08 02:40:28,332 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-3) [3a60121b] Installation of 10.70.36.35. Executing installation stage. (Stage: Connecting to Host) >2012-11-08 02:40:28,407 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-3) [3a60121b] Installation of 10.70.36.35. Received message: <BSTRAP component='RHEV_INSTALL' status='OK' message='Connected to Host 10.70.36.35 with SSH key fingerprint: 1a:00:b4:c0:0a:71:98:17:56:54:e6:00:1a:6d:dd:5d'/>. FYI. (Stage: Connecting to Host) >2012-11-08 02:40:28,457 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-3) [3a60121b] Installation of 10.70.36.35. Successfully connected to server ssh. (Stage: Connecting to Host) >2012-11-08 02:40:28,457 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-3) [3a60121b] Installation of 10.70.36.35. Executing installation stage. (Stage: Get the unique vds id) >2012-11-08 02:40:28,457 INFO [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-4-thread-3) SSH execute 10.70.36.35 'IDFILE=/etc/vdsm/vdsm.id; if [ -r "${IDFILE}" ]; then cat "${IDFILE}"; else UUID="$(dmidecode -s system-uuid 2> /dev/null | sed -e 's/.*Not.*//' )"; if [ -z "${UUID}" ]; then UUID="$(uuidgen 2> /dev/null)" && mkdir -p "$(dirname "${IDFILE}")" && echo "${UUID}" > "${IDFILE}" && chmod 0644 "${IDFILE}"; fi; [ -n "${UUID}" ] && echo "${UUID}"; fi' >2012-11-08 02:40:28,472 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-6) Installation of 10.70.36.35. Received message: 00000000-0000-0000-0000-0025907C2F76. FYI. (Stage: Get the unique vds id) >2012-11-08 02:40:28,494 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-6) Installation of 10.70.36.35. Assigning unique id 00000000-0000-0000-0000-0025907C2F76 to Host. (Stage: Get the unique vds id) >2012-11-08 02:40:28,503 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-3) [3a60121b] Installation of 10.70.36.35. Executing installation stage. (Stage: Upload Installation script to Host) >2012-11-08 02:40:28,504 WARN [org.ovirt.engine.core.dal.dbbroker.generic.DBConfigUtils] (pool-4-thread-3) [3a60121b] Adding new value to configuration cache. >2012-11-08 02:40:28,504 WARN [org.ovirt.engine.core.dal.dbbroker.generic.DBConfigUtils] (pool-4-thread-3) [3a60121b] Didnt find the value of SSHKeyAlias in DB for version general - using default: ovirt-engine >2012-11-08 02:40:28,505 INFO [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-4-thread-3) SSH execute 10.70.36.35 'cat > '/tmp/ovirt-id_rsa_e761a1bf-19ba-4647-9092-2b979f1b762e'' >2012-11-08 02:40:28,590 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-3) [3a60121b] Installation of 10.70.36.35. Executing installation stage. (Stage: Running first installation script on Host) >2012-11-08 02:40:28,591 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-3) [3a60121b] Installation of 10.70.36.35. Sending SSH Command umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar -C "${MYTMP}" --no-same-permissions -o -x && "${MYTMP}"/setup -c 'ssl=true;management_port=54321' -O 'lab.eng.blr.redhat.com' -t 2012-11-08T07:40:28 -S /tmp/ovirt-id_rsa_e761a1bf-19ba-4647-9092-2b979f1b762e -p 80 -b -V -B rhevm -g http://guido.lab.eng.blr.redhat.com:80/Components/vds/ http://guido.lab.eng.blr.redhat.com:80/Components/vds/ 10.70.36.35 e761a1bf-19ba-4647-9092-2b979f1b762e False < /var/cache/ovirt-engine/vdsm-bootstrap-2.tar. (Stage: Running first installation script on Host) >2012-11-08 02:40:28,612 INFO [org.ovirt.engine.core.utils.hostinstall.CachedTar] (pool-4-thread-3) Tarball '/var/cache/ovirt-engine/vdsm-bootstrap-2.tar' is out of date, re-creating >2012-11-08 02:40:28,652 INFO [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-4-thread-3) SSH execute 10.70.36.35 'umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar -C "${MYTMP}" --no-same-permissions -o -x && "${MYTMP}"/setup -c 'ssl=true;management_port=54321' -O 'lab.eng.blr.redhat.com' -t 2012-11-08T07:40:28 -S /tmp/ovirt-id_rsa_e761a1bf-19ba-4647-9092-2b979f1b762e -p 80 -b -V -B rhevm -g http://guido.lab.eng.blr.redhat.com:80/Components/vds/ http://guido.lab.eng.blr.redhat.com:80/Components/vds/ 10.70.36.35 e761a1bf-19ba-4647-9092-2b979f1b762e False' >2012-11-08 02:40:28,724 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-6) Installation of 10.70.36.35. Received message: <BSTRAP component='INSTALLER' status='OK' message='Test platform succeeded'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 02:40:29,823 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-6) Installation of 10.70.36.35. Received message: <BSTRAP component='SET_SYSTEM_TIME' status='OK' message='setSystemTime ended successfully'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 02:40:30,241 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-6) Installation of 10.70.36.35. Received message: <BSTRAP component='RHN_REGISTRATION' status='FAIL' message='Unable to fetch vdsm package. Please check if host is registered to RHN, Satellite or other yum repository'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 02:40:30,259 ERROR [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-6) Installation of 10.70.36.35. Received message: <BSTRAP component='RHEV_INSTALL' status='FAIL'/>. Error occured. (Stage: Running first installation script on Host) >2012-11-08 02:40:30,261 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-21) Initializing Host: rhs-client11.lab.eng.blr.redhat.com >2012-11-08 02:40:30,276 ERROR [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-4-thread-3) SSH error running command 10.70.36.35:'umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar -C "${MYTMP}" --no-same-permissions -o -x && "${MYTMP}"/setup -c 'ssl=true;management_port=54321' -O 'lab.eng.blr.redhat.com' -t 2012-11-08T07:40:28 -S /tmp/ovirt-id_rsa_e761a1bf-19ba-4647-9092-2b979f1b762e -p 80 -b -V -B rhevm -g http://guido.lab.eng.blr.redhat.com:80/Components/vds/ http://guido.lab.eng.blr.redhat.com:80/Components/vds/ 10.70.36.35 e761a1bf-19ba-4647-9092-2b979f1b762e False': java.io.IOException: Command returned failure code 1 during SSH session 10.70.36.35:22' 'umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar -C "${MYTMP}" --no-same-permissions -o -x && "${MYTMP}"/setup -c 'ssl=true;management_port=54321' -O 'lab.eng.blr.redhat.com' -t 2012-11-08T07:40:28 -S /tmp/ovirt-id_rsa_e761a1bf-19ba-4647-9092-2b979f1b762e -p 80 -b -V -B rhevm -g http://guido.lab.eng.blr.redhat.com:80/Components/vds/ http://guido.lab.eng.blr.redhat.com:80/Components/vds/ 10.70.36.35 e761a1bf-19ba-4647-9092-2b979f1b762e False' > at org.ovirt.engine.core.utils.ssh.SSHClient.executeCommand(SSHClient.java:442) [engine-utils.jar:] > at org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH.executeCommand(VdsInstallerSSH.java:387) [engine-utils.jar:] > at org.ovirt.engine.core.bll.VdsInstaller.runBootstrapCommand(VdsInstaller.java:217) [engine-bll.jar:] > at org.ovirt.engine.core.bll.VdsInstaller.RunStage(VdsInstaller.java:341) [engine-bll.jar:] > at org.ovirt.engine.core.bll.VdsInstaller.Install(VdsInstaller.java:280) [engine-bll.jar:] > at org.ovirt.engine.core.bll.InstallVdsCommand.executeCommand(InstallVdsCommand.java:110) [engine-bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:825) [engine-bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:916) [engine-bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1300) [engine-bll.jar:] > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:168) [engine-utils.jar:] > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:107) [engine-utils.jar:] > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:931) [engine-bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:285) [engine-bll.jar:] > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:336) [engine-bll.jar:] > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:593) [engine-bll.jar:] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_09-icedtea] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_09-icedtea] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_09-icedtea] > at java.lang.reflect.Method.invoke(Method.java:601) [rt.jar:1.7.0_09-icedtea] > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) [jboss-as-ee.jar:7.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:36) [jboss-as-ee.jar:7.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) [jboss-as-ejb3.jar:7.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) [jboss-as-ee.jar:7.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:53) [jboss-as-ejb3.jar:7.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:210) [jboss-as-ejb3.jar:7.1.2.Final-redhat-1] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:362) [jboss-as-ejb3.jar:7.1.2.Final-redhat-1] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:193) [jboss-as-ejb3.jar:7.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [jboss-as-ejb3.jar:7.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:42) [jboss-as-ejb3.jar:7.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) [jboss-as-ejb3.jar:7.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) [jboss-as-ee.jar:7.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) [jboss-as-ee.jar:7.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) [jboss-as-ee.jar:7.1.2.Final-redhat-1] > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:176) [jboss-as-ee.jar:7.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) [jboss-as-ee.jar:7.1.2.Final-redhat-1] > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) [engine-bll.jar:] > at org.ovirt.engine.core.bll.AddVdsCommand$3.run(AddVdsCommand.java:156) [engine-bll.jar:] > at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalWrapperRunnable.run(ThreadPoolUtil.java:64) [engine-utils.jar:] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_09-icedtea] > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) [rt.jar:1.7.0_09-icedtea] > at java.util.concurrent.FutureTask.run(FutureTask.java:166) [rt.jar:1.7.0_09-icedtea] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [rt.jar:1.7.0_09-icedtea] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [rt.jar:1.7.0_09-icedtea] > at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_09-icedtea] > >2012-11-08 02:40:30,279 ERROR [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-3) [3a60121b] Installation of 10.70.36.35 has failed. Failure details: SSH command failed while executing at host '10.70.36.35', refer to logs for further information. (Stage: Running first installation script on Host) >2012-11-08 02:40:30,279 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-3) [3a60121b] VDS message: SSH command failed while executing at host '10.70.36.35', refer to logs for further information >2012-11-08 02:40:30,279 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-3) [3a60121b] Script ended, result is {1} >2012-11-08 02:40:30,287 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-4-thread-3) [3a60121b] After Installation pool-4-thread-3 >2012-11-08 02:40:30,311 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-3) [3a60121b] START, SetVdsStatusVDSCommand(HostName = rhs-client11.lab.eng.blr.redhat.com, HostId = 90b581c2-2977-11e2-82cd-52540035ed48, status=InstallFailed, nonOperationalReason=NONE), log id: 6a0fd448 >2012-11-08 02:40:30,339 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-3) [3a60121b] FINISH, SetVdsStatusVDSCommand, log id: 6a0fd448 >2012-11-08 02:41:07,467 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp-/127.0.0.1:8702-7) [32ad77e] Running command: UpdateVdsCommand internal: false. Entities affected : ID: 90b581c2-2977-11e2-82cd-52540035ed48 Type: VDS >2012-11-08 02:41:07,524 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp-/127.0.0.1:8702-7) [32ad77e] START, SetVdsStatusVDSCommand(HostName = rhs-client11.lab.eng.blr.redhat.com, HostId = 90b581c2-2977-11e2-82cd-52540035ed48, status=Installing, nonOperationalReason=NONE), log id: 3eb7480a >2012-11-08 02:41:07,545 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp-/127.0.0.1:8702-7) [32ad77e] FINISH, SetVdsStatusVDSCommand, log id: 3eb7480a >2012-11-08 02:41:07,575 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-4-thread-4) [48c50918] Running command: InstallVdsCommand internal: true. Entities affected : ID: 90b581c2-2977-11e2-82cd-52540035ed48 Type: VDS >2012-11-08 02:41:07,579 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-4-thread-4) [48c50918] Before Installation pool-4-thread-4 >2012-11-08 02:41:07,579 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-4) [48c50918] Installation of 10.70.36.35. Executing installation stage. (Stage: Starting Host installation) >2012-11-08 02:41:07,579 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-4) [48c50918] Installation of 10.70.36.35. Executing installation stage. (Stage: Connecting to Host) >2012-11-08 02:41:07,616 ERROR [org.ovirt.engine.core.bll.FencingExecutor] (ajp-/127.0.0.1:8702-7) [32ad77e] Failed to run Power Management command on Host rhs-client11.lab.eng.blr.redhat.com, no running proxy Host was found. >2012-11-08 02:41:07,662 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-4) [48c50918] Installation of 10.70.36.35. Received message: <BSTRAP component='RHEV_INSTALL' status='OK' message='Connected to Host 10.70.36.35 with SSH key fingerprint: 1a:00:b4:c0:0a:71:98:17:56:54:e6:00:1a:6d:dd:5d'/>. FYI. (Stage: Connecting to Host) >2012-11-08 02:41:07,702 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-4) [48c50918] Installation of 10.70.36.35. Successfully connected to server ssh. (Stage: Connecting to Host) >2012-11-08 02:41:07,702 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-4) [48c50918] Installation of 10.70.36.35. Executing installation stage. (Stage: Get the unique vds id) >2012-11-08 02:41:07,702 INFO [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-4-thread-4) SSH execute 10.70.36.35 'IDFILE=/etc/vdsm/vdsm.id; if [ -r "${IDFILE}" ]; then cat "${IDFILE}"; else UUID="$(dmidecode -s system-uuid 2> /dev/null | sed -e 's/.*Not.*//' )"; if [ -z "${UUID}" ]; then UUID="$(uuidgen 2> /dev/null)" && mkdir -p "$(dirname "${IDFILE}")" && echo "${UUID}" > "${IDFILE}" && chmod 0644 "${IDFILE}"; fi; [ -n "${UUID}" ] && echo "${UUID}"; fi' >2012-11-08 02:41:07,717 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-7) Installation of 10.70.36.35. Received message: 00000000-0000-0000-0000-0025907C2F76. FYI. (Stage: Get the unique vds id) >2012-11-08 02:41:07,731 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-7) Installation of 10.70.36.35. Assigning unique id 00000000-0000-0000-0000-0025907C2F76 to Host. (Stage: Get the unique vds id) >2012-11-08 02:41:07,734 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-4) [48c50918] Installation of 10.70.36.35. Executing installation stage. (Stage: Upload Installation script to Host) >2012-11-08 02:41:07,734 INFO [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-4-thread-4) SSH execute 10.70.36.35 'cat > '/tmp/ovirt-id_rsa_216fca3c-68a7-4e0e-98c5-3b15db3a0d9c'' >2012-11-08 02:41:07,818 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-4) [48c50918] Installation of 10.70.36.35. Executing installation stage. (Stage: Running first installation script on Host) >2012-11-08 02:41:07,818 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-4) [48c50918] Installation of 10.70.36.35. Sending SSH Command umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar -C "${MYTMP}" --no-same-permissions -o -x && "${MYTMP}"/setup -c 'ssl=true;management_port=54321' -O 'lab.eng.blr.redhat.com' -t 2012-11-08T07:41:07 -S /tmp/ovirt-id_rsa_216fca3c-68a7-4e0e-98c5-3b15db3a0d9c -p 80 -b -V -B rhevm -g http://guido.lab.eng.blr.redhat.com:80/Components/vds/ http://guido.lab.eng.blr.redhat.com:80/Components/vds/ 10.70.36.35 216fca3c-68a7-4e0e-98c5-3b15db3a0d9c False < /var/cache/ovirt-engine/vdsm-bootstrap-2.tar. (Stage: Running first installation script on Host) >2012-11-08 02:41:07,818 INFO [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-4-thread-4) SSH execute 10.70.36.35 'umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar -C "${MYTMP}" --no-same-permissions -o -x && "${MYTMP}"/setup -c 'ssl=true;management_port=54321' -O 'lab.eng.blr.redhat.com' -t 2012-11-08T07:41:07 -S /tmp/ovirt-id_rsa_216fca3c-68a7-4e0e-98c5-3b15db3a0d9c -p 80 -b -V -B rhevm -g http://guido.lab.eng.blr.redhat.com:80/Components/vds/ http://guido.lab.eng.blr.redhat.com:80/Components/vds/ 10.70.36.35 216fca3c-68a7-4e0e-98c5-3b15db3a0d9c False' >2012-11-08 02:41:07,917 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-7) Installation of 10.70.36.35. Received message: <BSTRAP component='INSTALLER' status='OK' message='Test platform succeeded'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 02:41:08,786 WARN [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp-/127.0.0.1:8702-3) [2be4ac8a] CanDoAction of action UpdateVds failed. Reasons:VDS_CANNOT_INSTALL_STATUS_ILLEGAL,VAR__ACTION__UPDATE,VAR__TYPE__HOST >2012-11-08 02:41:10,003 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-7) Installation of 10.70.36.35. Received message: <BSTRAP component='SET_SYSTEM_TIME' status='OK' message='setSystemTime ended successfully'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 02:41:10,139 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-7) Installation of 10.70.36.35. Received message: <BSTRAP component='RHN_REGISTRATION' status='FAIL' message='Unable to fetch vdsm package. Please check if host is registered to RHN, Satellite or other yum repository'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 02:41:10,158 ERROR [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-7) Installation of 10.70.36.35. Received message: <BSTRAP component='RHEV_INSTALL' status='FAIL'/>. Error occured. (Stage: Running first installation script on Host) >2012-11-08 02:41:10,169 ERROR [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-4-thread-4) SSH error running command 10.70.36.35:'umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar -C "${MYTMP}" --no-same-permissions -o -x && "${MYTMP}"/setup -c 'ssl=true;management_port=54321' -O 'lab.eng.blr.redhat.com' -t 2012-11-08T07:41:07 -S /tmp/ovirt-id_rsa_216fca3c-68a7-4e0e-98c5-3b15db3a0d9c -p 80 -b -V -B rhevm -g http://guido.lab.eng.blr.redhat.com:80/Components/vds/ http://guido.lab.eng.blr.redhat.com:80/Components/vds/ 10.70.36.35 216fca3c-68a7-4e0e-98c5-3b15db3a0d9c False': java.io.IOException: Command returned failure code 1 during SSH session 10.70.36.35:22' 'umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar -C "${MYTMP}" --no-same-permissions -o -x && "${MYTMP}"/setup -c 'ssl=true;management_port=54321' -O 'lab.eng.blr.redhat.com' -t 2012-11-08T07:41:07 -S /tmp/ovirt-id_rsa_216fca3c-68a7-4e0e-98c5-3b15db3a0d9c -p 80 -b -V -B rhevm -g http://guido.lab.eng.blr.redhat.com:80/Components/vds/ http://guido.lab.eng.blr.redhat.com:80/Components/vds/ 10.70.36.35 216fca3c-68a7-4e0e-98c5-3b15db3a0d9c False' > at org.ovirt.engine.core.utils.ssh.SSHClient.executeCommand(SSHClient.java:442) [engine-utils.jar:] > at org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH.executeCommand(VdsInstallerSSH.java:387) [engine-utils.jar:] > at org.ovirt.engine.core.bll.VdsInstaller.runBootstrapCommand(VdsInstaller.java:217) [engine-bll.jar:] > at org.ovirt.engine.core.bll.VdsInstaller.RunStage(VdsInstaller.java:341) [engine-bll.jar:] > at org.ovirt.engine.core.bll.VdsInstaller.Install(VdsInstaller.java:280) [engine-bll.jar:] > at org.ovirt.engine.core.bll.InstallVdsCommand.executeCommand(InstallVdsCommand.java:110) [engine-bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:825) [engine-bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:916) [engine-bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1300) [engine-bll.jar:] > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:168) [engine-utils.jar:] > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:107) [engine-utils.jar:] > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:931) [engine-bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:285) [engine-bll.jar:] > at org.ovirt.engine.core.bll.MultipleActionsRunner.executeValidatedCommands(MultipleActionsRunner.java:182) [engine-bll.jar:] > at org.ovirt.engine.core.bll.MultipleActionsRunner.RunCommands(MultipleActionsRunner.java:162) [engine-bll.jar:] > at org.ovirt.engine.core.bll.MultipleActionsRunner$1.run(MultipleActionsRunner.java:84) [engine-bll.jar:] > at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalWrapperRunnable.run(ThreadPoolUtil.java:64) [engine-utils.jar:] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_09-icedtea] > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) [rt.jar:1.7.0_09-icedtea] > at java.util.concurrent.FutureTask.run(FutureTask.java:166) [rt.jar:1.7.0_09-icedtea] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [rt.jar:1.7.0_09-icedtea] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [rt.jar:1.7.0_09-icedtea] > at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_09-icedtea] > >2012-11-08 02:41:10,170 ERROR [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-4) [48c50918] Installation of 10.70.36.35 has failed. Failure details: SSH command failed while executing at host '10.70.36.35', refer to logs for further information. (Stage: Running first installation script on Host) >2012-11-08 02:41:10,170 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-4) [48c50918] VDS message: SSH command failed while executing at host '10.70.36.35', refer to logs for further information >2012-11-08 02:41:10,171 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-4) [48c50918] Script ended, result is {1} >2012-11-08 02:41:10,171 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-4-thread-4) [48c50918] After Installation pool-4-thread-4 >2012-11-08 02:41:10,182 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-4) [48c50918] START, SetVdsStatusVDSCommand(HostName = rhs-client11.lab.eng.blr.redhat.com, HostId = 90b581c2-2977-11e2-82cd-52540035ed48, status=InstallFailed, nonOperationalReason=NONE), log id: 3bc42d6f >2012-11-08 02:41:10,200 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-4) [48c50918] FINISH, SetVdsStatusVDSCommand, log id: 3bc42d6f >2012-11-08 02:42:31,351 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp-/127.0.0.1:8702-5) [bcc6c8b] Running command: UpdateVdsCommand internal: false. Entities affected : ID: 90b581c2-2977-11e2-82cd-52540035ed48 Type: VDS >2012-11-08 02:42:31,371 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp-/127.0.0.1:8702-5) [bcc6c8b] START, SetVdsStatusVDSCommand(HostName = rhs-client11.lab.eng.blr.redhat.com, HostId = 90b581c2-2977-11e2-82cd-52540035ed48, status=Installing, nonOperationalReason=NONE), log id: 2c7b643d >2012-11-08 02:42:31,384 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp-/127.0.0.1:8702-5) [bcc6c8b] FINISH, SetVdsStatusVDSCommand, log id: 2c7b643d >2012-11-08 02:42:31,396 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-4-thread-5) [3de375a9] Running command: InstallVdsCommand internal: true. Entities affected : ID: 90b581c2-2977-11e2-82cd-52540035ed48 Type: VDS >2012-11-08 02:42:31,399 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-4-thread-5) [3de375a9] Before Installation pool-4-thread-5 >2012-11-08 02:42:31,399 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-5) [3de375a9] Installation of 10.70.36.35. Executing installation stage. (Stage: Starting Host installation) >2012-11-08 02:42:31,399 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-5) [3de375a9] Installation of 10.70.36.35. Executing installation stage. (Stage: Connecting to Host) >2012-11-08 02:42:31,428 ERROR [org.ovirt.engine.core.bll.FencingExecutor] (ajp-/127.0.0.1:8702-5) [bcc6c8b] Failed to run Power Management command on Host rhs-client11.lab.eng.blr.redhat.com, no running proxy Host was found. >2012-11-08 02:42:31,475 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-5) [3de375a9] Installation of 10.70.36.35. Received message: <BSTRAP component='RHEV_INSTALL' status='OK' message='Connected to Host 10.70.36.35 with SSH key fingerprint: 1a:00:b4:c0:0a:71:98:17:56:54:e6:00:1a:6d:dd:5d'/>. FYI. (Stage: Connecting to Host) >2012-11-08 02:42:31,510 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-5) [3de375a9] Installation of 10.70.36.35. Successfully connected to server ssh. (Stage: Connecting to Host) >2012-11-08 02:42:31,510 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-5) [3de375a9] Installation of 10.70.36.35. Executing installation stage. (Stage: Get the unique vds id) >2012-11-08 02:42:31,510 INFO [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-4-thread-5) SSH execute 10.70.36.35 'IDFILE=/etc/vdsm/vdsm.id; if [ -r "${IDFILE}" ]; then cat "${IDFILE}"; else UUID="$(dmidecode -s system-uuid 2> /dev/null | sed -e 's/.*Not.*//' )"; if [ -z "${UUID}" ]; then UUID="$(uuidgen 2> /dev/null)" && mkdir -p "$(dirname "${IDFILE}")" && echo "${UUID}" > "${IDFILE}" && chmod 0644 "${IDFILE}"; fi; [ -n "${UUID}" ] && echo "${UUID}"; fi' >2012-11-08 02:42:31,525 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-11) Installation of 10.70.36.35. Received message: 00000000-0000-0000-0000-0025907C2F76. FYI. (Stage: Get the unique vds id) >2012-11-08 02:42:31,540 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-11) Installation of 10.70.36.35. Assigning unique id 00000000-0000-0000-0000-0025907C2F76 to Host. (Stage: Get the unique vds id) >2012-11-08 02:42:31,543 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-5) [3de375a9] Installation of 10.70.36.35. Executing installation stage. (Stage: Upload Installation script to Host) >2012-11-08 02:42:31,543 INFO [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-4-thread-5) SSH execute 10.70.36.35 'cat > '/tmp/ovirt-id_rsa_482535bf-5efb-47fc-a517-476250f54e3a'' >2012-11-08 02:42:31,631 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-5) [3de375a9] Installation of 10.70.36.35. Executing installation stage. (Stage: Running first installation script on Host) >2012-11-08 02:42:31,631 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-5) [3de375a9] Installation of 10.70.36.35. Sending SSH Command umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar -C "${MYTMP}" --no-same-permissions -o -x && "${MYTMP}"/setup -c 'ssl=true;management_port=54321' -O 'lab.eng.blr.redhat.com' -t 2012-11-08T07:42:31 -S /tmp/ovirt-id_rsa_482535bf-5efb-47fc-a517-476250f54e3a -p 80 -b -V -B rhevm -g http://guido.lab.eng.blr.redhat.com:80/Components/vds/ http://guido.lab.eng.blr.redhat.com:80/Components/vds/ 10.70.36.35 482535bf-5efb-47fc-a517-476250f54e3a False < /var/cache/ovirt-engine/vdsm-bootstrap-2.tar. (Stage: Running first installation script on Host) >2012-11-08 02:42:31,632 INFO [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-4-thread-5) SSH execute 10.70.36.35 'umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar -C "${MYTMP}" --no-same-permissions -o -x && "${MYTMP}"/setup -c 'ssl=true;management_port=54321' -O 'lab.eng.blr.redhat.com' -t 2012-11-08T07:42:31 -S /tmp/ovirt-id_rsa_482535bf-5efb-47fc-a517-476250f54e3a -p 80 -b -V -B rhevm -g http://guido.lab.eng.blr.redhat.com:80/Components/vds/ http://guido.lab.eng.blr.redhat.com:80/Components/vds/ 10.70.36.35 482535bf-5efb-47fc-a517-476250f54e3a False' >2012-11-08 02:42:31,724 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-11) Installation of 10.70.36.35. Received message: <BSTRAP component='INSTALLER' status='OK' message='Test platform succeeded'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 02:42:32,810 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-11) Installation of 10.70.36.35. Received message: <BSTRAP component='SET_SYSTEM_TIME' status='OK' message='setSystemTime ended successfully'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 02:42:32,946 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-11) Installation of 10.70.36.35. Received message: <BSTRAP component='RHN_REGISTRATION' status='FAIL' message='Unable to fetch vdsm package. Please check if host is registered to RHN, Satellite or other yum repository'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 02:42:32,964 ERROR [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-11) Installation of 10.70.36.35. Received message: <BSTRAP component='RHEV_INSTALL' status='FAIL'/>. Error occured. (Stage: Running first installation script on Host) >2012-11-08 02:42:32,979 ERROR [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-4-thread-5) SSH error running command 10.70.36.35:'umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar -C "${MYTMP}" --no-same-permissions -o -x && "${MYTMP}"/setup -c 'ssl=true;management_port=54321' -O 'lab.eng.blr.redhat.com' -t 2012-11-08T07:42:31 -S /tmp/ovirt-id_rsa_482535bf-5efb-47fc-a517-476250f54e3a -p 80 -b -V -B rhevm -g http://guido.lab.eng.blr.redhat.com:80/Components/vds/ http://guido.lab.eng.blr.redhat.com:80/Components/vds/ 10.70.36.35 482535bf-5efb-47fc-a517-476250f54e3a False': java.io.IOException: Command returned failure code 1 during SSH session 10.70.36.35:22' 'umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar -C "${MYTMP}" --no-same-permissions -o -x && "${MYTMP}"/setup -c 'ssl=true;management_port=54321' -O 'lab.eng.blr.redhat.com' -t 2012-11-08T07:42:31 -S /tmp/ovirt-id_rsa_482535bf-5efb-47fc-a517-476250f54e3a -p 80 -b -V -B rhevm -g http://guido.lab.eng.blr.redhat.com:80/Components/vds/ http://guido.lab.eng.blr.redhat.com:80/Components/vds/ 10.70.36.35 482535bf-5efb-47fc-a517-476250f54e3a False' > at org.ovirt.engine.core.utils.ssh.SSHClient.executeCommand(SSHClient.java:442) [engine-utils.jar:] > at org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH.executeCommand(VdsInstallerSSH.java:387) [engine-utils.jar:] > at org.ovirt.engine.core.bll.VdsInstaller.runBootstrapCommand(VdsInstaller.java:217) [engine-bll.jar:] > at org.ovirt.engine.core.bll.VdsInstaller.RunStage(VdsInstaller.java:341) [engine-bll.jar:] > at org.ovirt.engine.core.bll.VdsInstaller.Install(VdsInstaller.java:280) [engine-bll.jar:] > at org.ovirt.engine.core.bll.InstallVdsCommand.executeCommand(InstallVdsCommand.java:110) [engine-bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:825) [engine-bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:916) [engine-bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1300) [engine-bll.jar:] > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:168) [engine-utils.jar:] > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:107) [engine-utils.jar:] > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:931) [engine-bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:285) [engine-bll.jar:] > at org.ovirt.engine.core.bll.MultipleActionsRunner.executeValidatedCommands(MultipleActionsRunner.java:182) [engine-bll.jar:] > at org.ovirt.engine.core.bll.MultipleActionsRunner.RunCommands(MultipleActionsRunner.java:162) [engine-bll.jar:] > at org.ovirt.engine.core.bll.MultipleActionsRunner$1.run(MultipleActionsRunner.java:84) [engine-bll.jar:] > at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalWrapperRunnable.run(ThreadPoolUtil.java:64) [engine-utils.jar:] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_09-icedtea] > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) [rt.jar:1.7.0_09-icedtea] > at java.util.concurrent.FutureTask.run(FutureTask.java:166) [rt.jar:1.7.0_09-icedtea] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [rt.jar:1.7.0_09-icedtea] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [rt.jar:1.7.0_09-icedtea] > at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_09-icedtea] > >2012-11-08 02:42:32,979 ERROR [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-5) [3de375a9] Installation of 10.70.36.35 has failed. Failure details: SSH command failed while executing at host '10.70.36.35', refer to logs for further information. (Stage: Running first installation script on Host) >2012-11-08 02:42:32,980 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-5) [3de375a9] VDS message: SSH command failed while executing at host '10.70.36.35', refer to logs for further information >2012-11-08 02:42:32,980 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-5) [3de375a9] Script ended, result is {1} >2012-11-08 02:42:32,980 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-4-thread-5) [3de375a9] After Installation pool-4-thread-5 >2012-11-08 02:42:32,993 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-5) [3de375a9] START, SetVdsStatusVDSCommand(HostName = rhs-client11.lab.eng.blr.redhat.com, HostId = 90b581c2-2977-11e2-82cd-52540035ed48, status=InstallFailed, nonOperationalReason=NONE), log id: 6c9b07ac >2012-11-08 02:42:33,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-5) [3de375a9] FINISH, SetVdsStatusVDSCommand, log id: 6c9b07ac >2012-11-08 02:42:59,080 INFO [org.ovirt.engine.core.bll.MaintananceNumberOfVdssCommand] (pool-4-thread-6) [8204dee] Running command: MaintananceNumberOfVdssCommand internal: false. Entities affected : ID: 90b581c2-2977-11e2-82cd-52540035ed48 Type: VDS >2012-11-08 02:42:59,093 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-6) [8204dee] START, SetVdsStatusVDSCommand(HostName = rhs-client11.lab.eng.blr.redhat.com, HostId = 90b581c2-2977-11e2-82cd-52540035ed48, status=PreparingForMaintenance, nonOperationalReason=NONE), log id: 1fe2a0ed >2012-11-08 02:42:59,106 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-6) [8204dee] FINISH, SetVdsStatusVDSCommand, log id: 1fe2a0ed >2012-11-08 02:42:59,151 INFO [org.ovirt.engine.core.bll.MaintananceVdsCommand] (pool-4-thread-6) [8204dee] Running command: MaintananceVdsCommand internal: true. Entities affected : ID: 90b581c2-2977-11e2-82cd-52540035ed48 Type: VDS >2012-11-08 02:42:59,547 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-33) XML RPC error in command GetStatsVDS ( HostName = rhs-client11.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SSLException: Unrecognized SSL message, plaintext connection? >2012-11-08 02:42:59,553 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-33) vds::refreshVdsStats Failed getVdsStats, vds = 90b581c2-2977-11e2-82cd-52540035ed48 : rhs-client11.lab.eng.blr.redhat.com, error = VDSNetworkException: VDSNetworkException: >2012-11-08 02:42:59,560 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-33) ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = 90b581c2-2977-11e2-82cd-52540035ed48 : rhs-client11.lab.eng.blr.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-11-08 02:43:01,622 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-34) XML RPC error in command GetStatsVDS ( HostName = rhs-client11.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SSLException: Unrecognized SSL message, plaintext connection? >2012-11-08 02:43:01,623 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-34) vds::refreshVdsStats Failed getVdsStats, vds = 90b581c2-2977-11e2-82cd-52540035ed48 : rhs-client11.lab.eng.blr.redhat.com, error = VDSNetworkException: VDSNetworkException: >2012-11-08 02:43:01,623 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-34) ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = 90b581c2-2977-11e2-82cd-52540035ed48 : rhs-client11.lab.eng.blr.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-11-08 02:43:03,666 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-35) XML RPC error in command GetStatsVDS ( HostName = rhs-client11.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SSLException: Unrecognized SSL message, plaintext connection? >2012-11-08 02:43:03,666 ERROR [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-35) VDS::handleNetworkException Server failed to respond, vds_id = 90b581c2-2977-11e2-82cd-52540035ed48, vds_name = rhs-client11.lab.eng.blr.redhat.com, error = VDSNetworkException: >2012-11-08 02:43:03,692 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-35) vds::refreshVdsStats Failed getVdsStats, vds = 90b581c2-2977-11e2-82cd-52540035ed48 : rhs-client11.lab.eng.blr.redhat.com, error = VDSNetworkException: VDSNetworkException: >2012-11-08 02:43:03,692 INFO [org.ovirt.engine.core.bll.VdsEventListener] (pool-4-thread-10) ResourceManager::vdsNotResponding entered for Host 90b581c2-2977-11e2-82cd-52540035ed48, 10.70.36.35 >2012-11-08 02:43:03,743 INFO [org.ovirt.engine.core.bll.FencingExecutor] (pool-4-thread-10) Atempt 1 to find fencing proxy host failed... >2012-11-08 02:43:05,745 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-36) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-client11.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SSLException: Unrecognized SSL message, plaintext connection? >2012-11-08 02:43:06,421 ERROR [org.ovirt.engine.core.bll.FencingExecutor] (ajp-/127.0.0.1:8702-1) Failed to run Power Management command on Host , no running proxy Host was found. >2012-11-08 02:43:07,794 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-39) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-client11.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SSLException: Unrecognized SSL message, plaintext connection? >2012-11-08 02:43:09,840 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-40) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-client11.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SSLException: Unrecognized SSL message, plaintext connection? >2012-11-08 02:43:11,891 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-42) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-client11.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SSLException: Unrecognized SSL message, plaintext connection? >2012-11-08 02:43:12,938 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp-/127.0.0.1:8702-9) [30d13eb1] Running command: UpdateVdsCommand internal: false. Entities affected : ID: 90b581c2-2977-11e2-82cd-52540035ed48 Type: VDS >2012-11-08 02:43:13,945 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-41) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-client11.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SSLException: Unrecognized SSL message, plaintext connection? >2012-11-08 02:43:15,999 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-43) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-client11.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SSLException: Unrecognized SSL message, plaintext connection? >2012-11-08 02:43:18,055 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-46) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-client11.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SSLException: Unrecognized SSL message, plaintext connection? >2012-11-08 02:43:20,104 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-47) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-client11.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SSLException: Unrecognized SSL message, plaintext connection? >2012-11-08 02:43:22,143 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-48) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-client11.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SSLException: Unrecognized SSL message, plaintext connection? >2012-11-08 02:43:24,192 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-49) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-client11.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SSLException: Unrecognized SSL message, plaintext connection? >2012-11-08 02:43:26,237 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-52) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-client11.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SSLException: Unrecognized SSL message, plaintext connection? >2012-11-08 02:43:28,288 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-54) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-client11.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SSLException: Unrecognized SSL message, plaintext connection? >2012-11-08 02:43:28,911 INFO [org.ovirt.engine.core.bll.storage.FenceVdsManualyCommand] (ajp-/127.0.0.1:8702-11) Lock Acquired to object EngineLock [exclusiveLocks= key: 90b581c2-2977-11e2-82cd-52540035ed48 value: VDS_FENCE >, sharedLocks= ] >2012-11-08 02:43:28,923 INFO [org.ovirt.engine.core.bll.storage.FenceVdsManualyCommand] (pool-4-thread-23) [55f54621] Running command: FenceVdsManualyCommand internal: false. Entities affected : ID: 90b581c2-2977-11e2-82cd-52540035ed48 Type: VDS >2012-11-08 02:43:28,938 INFO [org.ovirt.engine.core.bll.ClearNonResponsiveVdsVmsCommand] (pool-4-thread-23) [154f9] Running command: ClearNonResponsiveVdsVmsCommand internal: true. Entities affected : ID: 90b581c2-2977-11e2-82cd-52540035ed48 Type: VDS >2012-11-08 02:43:28,950 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsVMsClearedVDSCommand] (pool-4-thread-23) [154f9] START, UpdateVdsVMsClearedVDSCommand(HostName = rhs-client11.lab.eng.blr.redhat.com, HostId = 90b581c2-2977-11e2-82cd-52540035ed48), log id: 2132d685 >2012-11-08 02:43:28,951 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsVMsClearedVDSCommand] (pool-4-thread-23) [154f9] FINISH, UpdateVdsVMsClearedVDSCommand, log id: 2132d685 >2012-11-08 02:43:28,958 INFO [org.ovirt.engine.core.bll.storage.FenceVdsManualyCommand] (pool-4-thread-23) [154f9] Lock freed to object EngineLock [exclusiveLocks= key: 90b581c2-2977-11e2-82cd-52540035ed48 value: VDS_FENCE >, sharedLocks= ] >2012-11-08 02:43:30,345 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-55) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-client11.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SSLException: Unrecognized SSL message, plaintext connection? >2012-11-08 02:43:32,380 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-56) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-client11.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SSLException: Unrecognized SSL message, plaintext connection? >2012-11-08 02:43:33,759 INFO [org.ovirt.engine.core.bll.FencingExecutor] (pool-4-thread-10) Atempt 2 to find fencing proxy host failed... >2012-11-08 02:43:34,444 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-57) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-client11.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SSLException: Unrecognized SSL message, plaintext connection? >2012-11-08 02:43:36,489 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-59) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-client11.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SSLException: Unrecognized SSL message, plaintext connection? >2012-11-08 02:43:38,528 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-61) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-client11.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SSLException: Unrecognized SSL message, plaintext connection? >2012-11-08 02:43:40,586 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-62) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-client11.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SSLException: Unrecognized SSL message, plaintext connection? >2012-11-08 02:43:42,641 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-63) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-client11.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SSLException: Unrecognized SSL message, plaintext connection? >2012-11-08 02:43:44,700 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-64) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-client11.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SSLException: Unrecognized SSL message, plaintext connection? >2012-11-08 02:43:46,751 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-67) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-client11.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SSLException: Unrecognized SSL message, plaintext connection? >2012-11-08 02:43:48,791 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-68) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-client11.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SSLException: Unrecognized SSL message, plaintext connection? >2012-11-08 02:43:50,847 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-69) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-client11.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SSLException: Unrecognized SSL message, plaintext connection? >2012-11-08 02:43:51,670 INFO [org.ovirt.engine.core.bll.MaintananceNumberOfVdssCommand] (pool-4-thread-35) [719cb965] Running command: MaintananceNumberOfVdssCommand internal: false. Entities affected : ID: 90b581c2-2977-11e2-82cd-52540035ed48 Type: VDS >2012-11-08 02:43:51,705 INFO [org.ovirt.engine.core.bll.MaintananceVdsCommand] (pool-4-thread-35) [719cb965] Running command: MaintananceVdsCommand internal: true. Entities affected : ID: 90b581c2-2977-11e2-82cd-52540035ed48 Type: VDS >2012-11-08 02:43:51,720 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-35) [719cb965] START, SetVdsStatusVDSCommand(HostName = rhs-client11.lab.eng.blr.redhat.com, HostId = 90b581c2-2977-11e2-82cd-52540035ed48, status=Maintenance, nonOperationalReason=NONE), log id: 4d54a792 >2012-11-08 02:43:51,731 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-35) [719cb965] FINISH, SetVdsStatusVDSCommand, log id: 4d54a792 >2012-11-08 02:43:56,923 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (ajp-/127.0.0.1:8702-6) Lock Acquired to object EngineLock [exclusiveLocks= key: 90b581c2-2977-11e2-82cd-52540035ed48 value: VDS >, sharedLocks= ] >2012-11-08 02:43:56,951 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (pool-4-thread-36) [af52749] Running command: ActivateVdsCommand internal: false. Entities affected : ID: 90b581c2-2977-11e2-82cd-52540035ed48 Type: VDS >2012-11-08 02:43:56,975 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-36) [af52749] START, SetVdsStatusVDSCommand(HostName = rhs-client11.lab.eng.blr.redhat.com, HostId = 90b581c2-2977-11e2-82cd-52540035ed48, status=Unassigned, nonOperationalReason=NONE), log id: 77668827 >2012-11-08 02:43:56,995 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-36) [af52749] FINISH, SetVdsStatusVDSCommand, log id: 77668827 >2012-11-08 02:43:57,009 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (pool-4-thread-36) [af52749] START, ActivateVdsVDSCommand(HostName = rhs-client11.lab.eng.blr.redhat.com, HostId = 90b581c2-2977-11e2-82cd-52540035ed48), log id: 2e331a55 >2012-11-08 02:43:57,041 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (pool-4-thread-36) [af52749] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-client11.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SSLException: Unrecognized SSL message, plaintext connection? >2012-11-08 02:43:57,042 ERROR [org.ovirt.engine.core.vdsbroker.VdsManager] (pool-4-thread-36) [af52749] VDS::handleNetworkException Server failed to respond, vds_id = 90b581c2-2977-11e2-82cd-52540035ed48, vds_name = rhs-client11.lab.eng.blr.redhat.com, error = VDSNetworkException: >2012-11-08 02:43:57,067 INFO [org.ovirt.engine.core.bll.VdsEventListener] (pool-4-thread-38) [af52749] ResourceManager::vdsNotResponding entered for Host 90b581c2-2977-11e2-82cd-52540035ed48, 10.70.36.35 >2012-11-08 02:43:57,070 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (pool-4-thread-36) [af52749] ResourceManager::activateVds - failed to get VDS = 90b581c2-2977-11e2-82cd-52540035ed48 capabilities with error: VDSNetworkException: . >2012-11-08 02:43:57,070 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (pool-4-thread-36) [af52749] ResourceManager::activateVds - failed to activate VDS = 90b581c2-2977-11e2-82cd-52540035ed48 >2012-11-08 02:43:57,093 WARN [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (pool-4-thread-36) [5df37fe6] CanDoAction of action HandleVdsVersion failed. Reasons:VDS_CANNOT_CHECK_VERSION_HOST_NON_RESPONSIVE >2012-11-08 02:43:57,093 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (pool-4-thread-36) [5df37fe6] FINISH, ActivateVdsVDSCommand, log id: 2e331a55 >2012-11-08 02:43:57,110 WARN [org.ovirt.engine.core.bll.VdsNotRespondingTreatmentCommand] (pool-4-thread-38) [2445daca] CanDoAction of action VdsNotRespondingTreatment failed. Reasons:VDS_FENCING_DISABLED >2012-11-08 02:43:57,128 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (pool-4-thread-36) [5df37fe6] Lock freed to object EngineLock [exclusiveLocks= key: 90b581c2-2977-11e2-82cd-52540035ed48 value: VDS >, sharedLocks= ] >2012-11-08 02:43:58,951 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-77) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-client11.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SSLException: Unrecognized SSL message, plaintext connection? >2012-11-08 02:44:01,012 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-78) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-client11.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SSLException: Unrecognized SSL message, plaintext connection? >2012-11-08 02:44:03,054 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-79) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-client11.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SSLException: Unrecognized SSL message, plaintext connection? >2012-11-08 02:44:03,789 INFO [org.ovirt.engine.core.bll.FencingExecutor] (pool-4-thread-10) Atempt 3 to find fencing proxy host failed... >2012-11-08 02:44:05,106 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-80) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-client11.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SSLException: Unrecognized SSL message, plaintext connection? >2012-11-08 02:44:07,170 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-83) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-client11.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SSLException: Unrecognized SSL message, plaintext connection? >2012-11-08 02:44:09,223 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-84) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-client11.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SSLException: Unrecognized SSL message, plaintext connection? >2012-11-08 02:44:11,261 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-85) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-client11.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SSLException: Unrecognized SSL message, plaintext connection? >2012-11-08 02:44:12,232 INFO [org.ovirt.engine.core.bll.MaintananceNumberOfVdssCommand] (pool-4-thread-46) [135191b] Running command: MaintananceNumberOfVdssCommand internal: false. Entities affected : ID: 90b581c2-2977-11e2-82cd-52540035ed48 Type: VDS >2012-11-08 02:44:12,292 INFO [org.ovirt.engine.core.bll.MaintananceVdsCommand] (pool-4-thread-46) [135191b] Running command: MaintananceVdsCommand internal: true. Entities affected : ID: 90b581c2-2977-11e2-82cd-52540035ed48 Type: VDS >2012-11-08 02:44:12,305 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-46) [135191b] START, SetVdsStatusVDSCommand(HostName = rhs-client11.lab.eng.blr.redhat.com, HostId = 90b581c2-2977-11e2-82cd-52540035ed48, status=Maintenance, nonOperationalReason=NONE), log id: 7e794da9 >2012-11-08 02:44:12,316 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-46) [135191b] FINISH, SetVdsStatusVDSCommand, log id: 7e794da9 >2012-11-08 02:44:19,254 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp-/127.0.0.1:8702-5) [33b0fbee] Running command: UpdateVdsCommand internal: false. Entities affected : ID: 90b581c2-2977-11e2-82cd-52540035ed48 Type: VDS >2012-11-08 02:44:19,285 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp-/127.0.0.1:8702-5) [33b0fbee] START, SetVdsStatusVDSCommand(HostName = rhs-client11.lab.eng.blr.redhat.com, HostId = 90b581c2-2977-11e2-82cd-52540035ed48, status=Installing, nonOperationalReason=NONE), log id: 7c7f79b7 >2012-11-08 02:44:19,298 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp-/127.0.0.1:8702-5) [33b0fbee] FINISH, SetVdsStatusVDSCommand, log id: 7c7f79b7 >2012-11-08 02:44:19,311 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-4-thread-47) [71348588] Running command: InstallVdsCommand internal: true. Entities affected : ID: 90b581c2-2977-11e2-82cd-52540035ed48 Type: VDS >2012-11-08 02:44:19,313 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-4-thread-47) [71348588] Before Installation pool-4-thread-47 >2012-11-08 02:44:19,313 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-47) [71348588] Installation of 10.70.36.35. Executing installation stage. (Stage: Starting Host installation) >2012-11-08 02:44:19,313 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-47) [71348588] Installation of 10.70.36.35. Executing installation stage. (Stage: Connecting to Host) >2012-11-08 02:44:19,387 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-47) [71348588] Installation of 10.70.36.35. Received message: <BSTRAP component='RHEV_INSTALL' status='OK' message='Connected to Host 10.70.36.35 with SSH key fingerprint: 1a:00:b4:c0:0a:71:98:17:56:54:e6:00:1a:6d:dd:5d'/>. FYI. (Stage: Connecting to Host) >2012-11-08 02:44:19,429 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-47) [71348588] Installation of 10.70.36.35. Successfully connected to server ssh. (Stage: Connecting to Host) >2012-11-08 02:44:19,429 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-47) [71348588] Installation of 10.70.36.35. Executing installation stage. (Stage: Get the unique vds id) >2012-11-08 02:44:19,429 INFO [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-4-thread-47) SSH execute 10.70.36.35 'IDFILE=/etc/vdsm/vdsm.id; if [ -r "${IDFILE}" ]; then cat "${IDFILE}"; else UUID="$(dmidecode -s system-uuid 2> /dev/null | sed -e 's/.*Not.*//' )"; if [ -z "${UUID}" ]; then UUID="$(uuidgen 2> /dev/null)" && mkdir -p "$(dirname "${IDFILE}")" && echo "${UUID}" > "${IDFILE}" && chmod 0644 "${IDFILE}"; fi; [ -n "${UUID}" ] && echo "${UUID}"; fi' >2012-11-08 02:44:19,445 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-15) Installation of 10.70.36.35. Received message: 00000000-0000-0000-0000-0025907C2F76. FYI. (Stage: Get the unique vds id) >2012-11-08 02:44:19,463 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-15) Installation of 10.70.36.35. Assigning unique id 00000000-0000-0000-0000-0025907C2F76 to Host. (Stage: Get the unique vds id) >2012-11-08 02:44:19,466 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-47) [71348588] Installation of 10.70.36.35. Executing installation stage. (Stage: Upload Installation script to Host) >2012-11-08 02:44:19,467 INFO [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-4-thread-47) SSH execute 10.70.36.35 'cat > '/tmp/ovirt-id_rsa_c6439512-cbad-4723-a53e-301c645dc4e6'' >2012-11-08 02:44:19,551 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-47) [71348588] Installation of 10.70.36.35. Executing installation stage. (Stage: Running first installation script on Host) >2012-11-08 02:44:19,551 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-47) [71348588] Installation of 10.70.36.35. Sending SSH Command umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar -C "${MYTMP}" --no-same-permissions -o -x && "${MYTMP}"/setup -c 'ssl=true;management_port=54321' -O 'lab.eng.blr.redhat.com' -t 2012-11-08T07:44:19 -S /tmp/ovirt-id_rsa_c6439512-cbad-4723-a53e-301c645dc4e6 -p 80 -b -V -B rhevm -g http://guido.lab.eng.blr.redhat.com:80/Components/vds/ http://guido.lab.eng.blr.redhat.com:80/Components/vds/ 10.70.36.35 c6439512-cbad-4723-a53e-301c645dc4e6 False < /var/cache/ovirt-engine/vdsm-bootstrap-2.tar. (Stage: Running first installation script on Host) >2012-11-08 02:44:19,552 INFO [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-4-thread-47) SSH execute 10.70.36.35 'umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar -C "${MYTMP}" --no-same-permissions -o -x && "${MYTMP}"/setup -c 'ssl=true;management_port=54321' -O 'lab.eng.blr.redhat.com' -t 2012-11-08T07:44:19 -S /tmp/ovirt-id_rsa_c6439512-cbad-4723-a53e-301c645dc4e6 -p 80 -b -V -B rhevm -g http://guido.lab.eng.blr.redhat.com:80/Components/vds/ http://guido.lab.eng.blr.redhat.com:80/Components/vds/ 10.70.36.35 c6439512-cbad-4723-a53e-301c645dc4e6 False' >2012-11-08 02:44:19,643 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-15) Installation of 10.70.36.35. Received message: <BSTRAP component='INSTALLER' status='OK' message='Test platform succeeded'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 02:44:20,729 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-15) Installation of 10.70.36.35. Received message: <BSTRAP component='SET_SYSTEM_TIME' status='OK' message='setSystemTime ended successfully'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 02:44:20,863 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-15) Installation of 10.70.36.35. Received message: <BSTRAP component='RHN_REGISTRATION' status='FAIL' message='Unable to fetch vdsm package. Please check if host is registered to RHN, Satellite or other yum repository'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 02:44:20,880 ERROR [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-15) Installation of 10.70.36.35. Received message: <BSTRAP component='RHEV_INSTALL' status='FAIL'/>. Error occured. (Stage: Running first installation script on Host) >2012-11-08 02:44:20,891 ERROR [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-4-thread-47) SSH error running command 10.70.36.35:'umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar -C "${MYTMP}" --no-same-permissions -o -x && "${MYTMP}"/setup -c 'ssl=true;management_port=54321' -O 'lab.eng.blr.redhat.com' -t 2012-11-08T07:44:19 -S /tmp/ovirt-id_rsa_c6439512-cbad-4723-a53e-301c645dc4e6 -p 80 -b -V -B rhevm -g http://guido.lab.eng.blr.redhat.com:80/Components/vds/ http://guido.lab.eng.blr.redhat.com:80/Components/vds/ 10.70.36.35 c6439512-cbad-4723-a53e-301c645dc4e6 False': java.io.IOException: Command returned failure code 1 during SSH session 10.70.36.35:22' 'umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar -C "${MYTMP}" --no-same-permissions -o -x && "${MYTMP}"/setup -c 'ssl=true;management_port=54321' -O 'lab.eng.blr.redhat.com' -t 2012-11-08T07:44:19 -S /tmp/ovirt-id_rsa_c6439512-cbad-4723-a53e-301c645dc4e6 -p 80 -b -V -B rhevm -g http://guido.lab.eng.blr.redhat.com:80/Components/vds/ http://guido.lab.eng.blr.redhat.com:80/Components/vds/ 10.70.36.35 c6439512-cbad-4723-a53e-301c645dc4e6 False' > at org.ovirt.engine.core.utils.ssh.SSHClient.executeCommand(SSHClient.java:442) [engine-utils.jar:] > at org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH.executeCommand(VdsInstallerSSH.java:387) [engine-utils.jar:] > at org.ovirt.engine.core.bll.VdsInstaller.runBootstrapCommand(VdsInstaller.java:217) [engine-bll.jar:] > at org.ovirt.engine.core.bll.VdsInstaller.RunStage(VdsInstaller.java:341) [engine-bll.jar:] > at org.ovirt.engine.core.bll.VdsInstaller.Install(VdsInstaller.java:280) [engine-bll.jar:] > at org.ovirt.engine.core.bll.InstallVdsCommand.executeCommand(InstallVdsCommand.java:110) [engine-bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:825) [engine-bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:916) [engine-bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1300) [engine-bll.jar:] > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:168) [engine-utils.jar:] > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:107) [engine-utils.jar:] > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:931) [engine-bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:285) [engine-bll.jar:] > at org.ovirt.engine.core.bll.MultipleActionsRunner.executeValidatedCommands(MultipleActionsRunner.java:182) [engine-bll.jar:] > at org.ovirt.engine.core.bll.MultipleActionsRunner.RunCommands(MultipleActionsRunner.java:162) [engine-bll.jar:] > at org.ovirt.engine.core.bll.MultipleActionsRunner$1.run(MultipleActionsRunner.java:84) [engine-bll.jar:] > at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalWrapperRunnable.run(ThreadPoolUtil.java:64) [engine-utils.jar:] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_09-icedtea] > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) [rt.jar:1.7.0_09-icedtea] > at java.util.concurrent.FutureTask.run(FutureTask.java:166) [rt.jar:1.7.0_09-icedtea] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [rt.jar:1.7.0_09-icedtea] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [rt.jar:1.7.0_09-icedtea] > at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_09-icedtea] > >2012-11-08 02:44:20,892 ERROR [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-47) [71348588] Installation of 10.70.36.35 has failed. Failure details: SSH command failed while executing at host '10.70.36.35', refer to logs for further information. (Stage: Running first installation script on Host) >2012-11-08 02:44:20,892 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-47) [71348588] VDS message: SSH command failed while executing at host '10.70.36.35', refer to logs for further information >2012-11-08 02:44:20,892 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-47) [71348588] Script ended, result is {1} >2012-11-08 02:44:20,892 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-4-thread-47) [71348588] After Installation pool-4-thread-47 >2012-11-08 02:44:20,903 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-47) [71348588] START, SetVdsStatusVDSCommand(HostName = rhs-client11.lab.eng.blr.redhat.com, HostId = 90b581c2-2977-11e2-82cd-52540035ed48, status=InstallFailed, nonOperationalReason=NONE), log id: 1ce5853a >2012-11-08 02:44:20,934 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-47) [71348588] FINISH, SetVdsStatusVDSCommand, log id: 1ce5853a >2012-11-08 02:44:33,789 ERROR [org.ovirt.engine.core.bll.FencingExecutor] (pool-4-thread-10) Failed to run Power Management command on Host rhs-client11.lab.eng.blr.redhat.com, no running proxy Host was found. >2012-11-08 02:44:33,790 ERROR [org.ovirt.engine.core.bll.VdsNotRespondingTreatmentCommand] (pool-4-thread-10) Failed to run Fence script on vds:rhs-client11.lab.eng.blr.redhat.com, VMs moved to UnKnown instead. >2012-11-08 02:44:33,790 WARN [org.ovirt.engine.core.bll.VdsNotRespondingTreatmentCommand] (pool-4-thread-10) CanDoAction of action VdsNotRespondingTreatment failed. Reasons:VDS_NO_VDS_PROXY_FOUND >2012-11-08 02:45:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-24) Autorecovering hosts is disabled, skipping >2012-11-08 02:45:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-24) Autorecovering storage domains is disabled, skipping >2012-11-08 02:45:21,370 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp-/127.0.0.1:8702-7) [2e98e1ad] Running command: UpdateVdsCommand internal: false. Entities affected : ID: 90b581c2-2977-11e2-82cd-52540035ed48 Type: VDS >2012-11-08 02:45:21,392 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp-/127.0.0.1:8702-7) [2e98e1ad] START, SetVdsStatusVDSCommand(HostName = rhs-client11.lab.eng.blr.redhat.com, HostId = 90b581c2-2977-11e2-82cd-52540035ed48, status=Installing, nonOperationalReason=NONE), log id: 28723091 >2012-11-08 02:45:21,404 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp-/127.0.0.1:8702-7) [2e98e1ad] FINISH, SetVdsStatusVDSCommand, log id: 28723091 >2012-11-08 02:45:21,417 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-4-thread-48) [7625ba57] Running command: InstallVdsCommand internal: true. Entities affected : ID: 90b581c2-2977-11e2-82cd-52540035ed48 Type: VDS >2012-11-08 02:45:21,420 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-4-thread-48) [7625ba57] Before Installation pool-4-thread-48 >2012-11-08 02:45:21,420 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-48) [7625ba57] Installation of 10.70.36.35. Executing installation stage. (Stage: Starting Host installation) >2012-11-08 02:45:21,507 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-48) [7625ba57] Installation of 10.70.36.35. Executing installation stage. (Stage: Connecting to Host) >2012-11-08 02:45:21,577 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-48) [7625ba57] Installation of 10.70.36.35. Received message: <BSTRAP component='RHEV_INSTALL' status='OK' message='Connected to Host 10.70.36.35 with SSH key fingerprint: 1a:00:b4:c0:0a:71:98:17:56:54:e6:00:1a:6d:dd:5d'/>. FYI. (Stage: Connecting to Host) >2012-11-08 02:45:21,622 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-48) [7625ba57] Installation of 10.70.36.35. Successfully connected to server ssh. (Stage: Connecting to Host) >2012-11-08 02:45:21,622 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-48) [7625ba57] Installation of 10.70.36.35. Executing installation stage. (Stage: Get the unique vds id) >2012-11-08 02:45:21,622 INFO [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-4-thread-48) SSH execute 10.70.36.35 'IDFILE=/etc/vdsm/vdsm.id; if [ -r "${IDFILE}" ]; then cat "${IDFILE}"; else UUID="$(dmidecode -s system-uuid 2> /dev/null | sed -e 's/.*Not.*//' )"; if [ -z "${UUID}" ]; then UUID="$(uuidgen 2> /dev/null)" && mkdir -p "$(dirname "${IDFILE}")" && echo "${UUID}" > "${IDFILE}" && chmod 0644 "${IDFILE}"; fi; [ -n "${UUID}" ] && echo "${UUID}"; fi' >2012-11-08 02:45:21,640 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-16) Installation of 10.70.36.35. Received message: 00000000-0000-0000-0000-0025907C2F76. FYI. (Stage: Get the unique vds id) >2012-11-08 02:45:21,654 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-16) Installation of 10.70.36.35. Assigning unique id 00000000-0000-0000-0000-0025907C2F76 to Host. (Stage: Get the unique vds id) >2012-11-08 02:45:21,657 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-48) [7625ba57] Installation of 10.70.36.35. Executing installation stage. (Stage: Upload Installation script to Host) >2012-11-08 02:45:21,657 INFO [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-4-thread-48) SSH execute 10.70.36.35 'cat > '/tmp/ovirt-id_rsa_c7dd1784-ad20-4287-a49b-9ebcaced867f'' >2012-11-08 02:45:21,742 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-48) [7625ba57] Installation of 10.70.36.35. Executing installation stage. (Stage: Running first installation script on Host) >2012-11-08 02:45:21,742 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-48) [7625ba57] Installation of 10.70.36.35. Sending SSH Command umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar -C "${MYTMP}" --no-same-permissions -o -x && "${MYTMP}"/setup -c 'ssl=true;management_port=54321' -O 'lab.eng.blr.redhat.com' -t 2012-11-08T07:45:21 -S /tmp/ovirt-id_rsa_c7dd1784-ad20-4287-a49b-9ebcaced867f -p 80 -b -V -B rhevm -g http://guido.lab.eng.blr.redhat.com:80/Components/vds/ http://guido.lab.eng.blr.redhat.com:80/Components/vds/ 10.70.36.35 c7dd1784-ad20-4287-a49b-9ebcaced867f False < /var/cache/ovirt-engine/vdsm-bootstrap-2.tar. (Stage: Running first installation script on Host) >2012-11-08 02:45:21,743 INFO [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-4-thread-48) SSH execute 10.70.36.35 'umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar -C "${MYTMP}" --no-same-permissions -o -x && "${MYTMP}"/setup -c 'ssl=true;management_port=54321' -O 'lab.eng.blr.redhat.com' -t 2012-11-08T07:45:21 -S /tmp/ovirt-id_rsa_c7dd1784-ad20-4287-a49b-9ebcaced867f -p 80 -b -V -B rhevm -g http://guido.lab.eng.blr.redhat.com:80/Components/vds/ http://guido.lab.eng.blr.redhat.com:80/Components/vds/ 10.70.36.35 c7dd1784-ad20-4287-a49b-9ebcaced867f False' >2012-11-08 02:45:21,837 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-16) Installation of 10.70.36.35. Received message: <BSTRAP component='INSTALLER' status='OK' message='Test platform succeeded'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 02:45:23,922 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-16) Installation of 10.70.36.35. Received message: <BSTRAP component='SET_SYSTEM_TIME' status='OK' message='setSystemTime ended successfully'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 02:45:24,056 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-16) Installation of 10.70.36.35. Received message: <BSTRAP component='RHN_REGISTRATION' status='FAIL' message='Unable to fetch vdsm package. Please check if host is registered to RHN, Satellite or other yum repository'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 02:45:24,073 ERROR [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-16) Installation of 10.70.36.35. Received message: <BSTRAP component='RHEV_INSTALL' status='FAIL'/>. Error occured. (Stage: Running first installation script on Host) >2012-11-08 02:45:24,091 ERROR [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-4-thread-48) SSH error running command 10.70.36.35:'umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar -C "${MYTMP}" --no-same-permissions -o -x && "${MYTMP}"/setup -c 'ssl=true;management_port=54321' -O 'lab.eng.blr.redhat.com' -t 2012-11-08T07:45:21 -S /tmp/ovirt-id_rsa_c7dd1784-ad20-4287-a49b-9ebcaced867f -p 80 -b -V -B rhevm -g http://guido.lab.eng.blr.redhat.com:80/Components/vds/ http://guido.lab.eng.blr.redhat.com:80/Components/vds/ 10.70.36.35 c7dd1784-ad20-4287-a49b-9ebcaced867f False': java.io.IOException: Command returned failure code 1 during SSH session 10.70.36.35:22' 'umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar -C "${MYTMP}" --no-same-permissions -o -x && "${MYTMP}"/setup -c 'ssl=true;management_port=54321' -O 'lab.eng.blr.redhat.com' -t 2012-11-08T07:45:21 -S /tmp/ovirt-id_rsa_c7dd1784-ad20-4287-a49b-9ebcaced867f -p 80 -b -V -B rhevm -g http://guido.lab.eng.blr.redhat.com:80/Components/vds/ http://guido.lab.eng.blr.redhat.com:80/Components/vds/ 10.70.36.35 c7dd1784-ad20-4287-a49b-9ebcaced867f False' > at org.ovirt.engine.core.utils.ssh.SSHClient.executeCommand(SSHClient.java:442) [engine-utils.jar:] > at org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH.executeCommand(VdsInstallerSSH.java:387) [engine-utils.jar:] > at org.ovirt.engine.core.bll.VdsInstaller.runBootstrapCommand(VdsInstaller.java:217) [engine-bll.jar:] > at org.ovirt.engine.core.bll.VdsInstaller.RunStage(VdsInstaller.java:341) [engine-bll.jar:] > at org.ovirt.engine.core.bll.VdsInstaller.Install(VdsInstaller.java:280) [engine-bll.jar:] > at org.ovirt.engine.core.bll.InstallVdsCommand.executeCommand(InstallVdsCommand.java:110) [engine-bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:825) [engine-bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:916) [engine-bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1300) [engine-bll.jar:] > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:168) [engine-utils.jar:] > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:107) [engine-utils.jar:] > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:931) [engine-bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:285) [engine-bll.jar:] > at org.ovirt.engine.core.bll.MultipleActionsRunner.executeValidatedCommands(MultipleActionsRunner.java:182) [engine-bll.jar:] > at org.ovirt.engine.core.bll.MultipleActionsRunner.RunCommands(MultipleActionsRunner.java:162) [engine-bll.jar:] > at org.ovirt.engine.core.bll.MultipleActionsRunner$1.run(MultipleActionsRunner.java:84) [engine-bll.jar:] > at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalWrapperRunnable.run(ThreadPoolUtil.java:64) [engine-utils.jar:] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_09-icedtea] > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) [rt.jar:1.7.0_09-icedtea] > at java.util.concurrent.FutureTask.run(FutureTask.java:166) [rt.jar:1.7.0_09-icedtea] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [rt.jar:1.7.0_09-icedtea] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [rt.jar:1.7.0_09-icedtea] > at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_09-icedtea] > >2012-11-08 02:45:24,091 ERROR [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-48) [7625ba57] Installation of 10.70.36.35 has failed. Failure details: SSH command failed while executing at host '10.70.36.35', refer to logs for further information. (Stage: Running first installation script on Host) >2012-11-08 02:45:24,091 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-48) [7625ba57] VDS message: SSH command failed while executing at host '10.70.36.35', refer to logs for further information >2012-11-08 02:45:24,091 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-48) [7625ba57] Script ended, result is {1} >2012-11-08 02:45:24,092 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-4-thread-48) [7625ba57] After Installation pool-4-thread-48 >2012-11-08 02:45:24,111 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-48) [7625ba57] START, SetVdsStatusVDSCommand(HostName = rhs-client11.lab.eng.blr.redhat.com, HostId = 90b581c2-2977-11e2-82cd-52540035ed48, status=InstallFailed, nonOperationalReason=NONE), log id: 6f58cea4 >2012-11-08 02:45:24,129 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-48) [7625ba57] FINISH, SetVdsStatusVDSCommand, log id: 6f58cea4 >2012-11-08 02:50:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-52) Autorecovering hosts is disabled, skipping >2012-11-08 02:50:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-52) Autorecovering storage domains is disabled, skipping >2012-11-08 02:55:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-77) Autorecovering hosts is disabled, skipping >2012-11-08 02:55:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-77) Autorecovering storage domains is disabled, skipping >2012-11-08 03:00:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-3) Autorecovering hosts is disabled, skipping >2012-11-08 03:00:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-3) Autorecovering storage domains is disabled, skipping >2012-11-08 03:05:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-30) Autorecovering hosts is disabled, skipping >2012-11-08 03:05:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-30) Autorecovering storage domains is disabled, skipping >2012-11-08 03:10:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-56) Autorecovering hosts is disabled, skipping >2012-11-08 03:10:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-56) Autorecovering storage domains is disabled, skipping >2012-11-08 03:15:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-83) Autorecovering hosts is disabled, skipping >2012-11-08 03:15:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-83) Autorecovering storage domains is disabled, skipping >2012-11-08 03:19:55,929 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-3) DbUserCacheManager::refreshAllUserData() - entered >2012-11-08 03:20:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-12) Autorecovering hosts is disabled, skipping >2012-11-08 03:20:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-12) Autorecovering storage domains is disabled, skipping >2012-11-08 03:25:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-39) Autorecovering hosts is disabled, skipping >2012-11-08 03:25:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-39) Autorecovering storage domains is disabled, skipping >2012-11-08 03:30:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-65) Autorecovering hosts is disabled, skipping >2012-11-08 03:30:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-65) Autorecovering storage domains is disabled, skipping >2012-11-08 03:35:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-91) Autorecovering hosts is disabled, skipping >2012-11-08 03:35:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-91) Autorecovering storage domains is disabled, skipping >2012-11-08 03:35:35,001 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (QuartzScheduler_Worker-15) AuditLogCleanupManager::deleteAgedOutAuditLogs - entered >2012-11-08 03:40:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-19) Autorecovering hosts is disabled, skipping >2012-11-08 03:40:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-19) Autorecovering storage domains is disabled, skipping >2012-11-08 03:45:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-45) Autorecovering hosts is disabled, skipping >2012-11-08 03:45:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-45) Autorecovering storage domains is disabled, skipping >2012-11-08 03:50:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-73) Autorecovering hosts is disabled, skipping >2012-11-08 03:50:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-73) Autorecovering storage domains is disabled, skipping >2012-11-08 03:55:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-99) Autorecovering hosts is disabled, skipping >2012-11-08 03:55:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-99) Autorecovering storage domains is disabled, skipping >2012-11-08 04:00:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-26) Autorecovering hosts is disabled, skipping >2012-11-08 04:00:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-26) Autorecovering storage domains is disabled, skipping >2012-11-08 04:05:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-52) Autorecovering hosts is disabled, skipping >2012-11-08 04:05:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-52) Autorecovering storage domains is disabled, skipping >2012-11-08 04:10:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-78) Autorecovering hosts is disabled, skipping >2012-11-08 04:10:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-78) Autorecovering storage domains is disabled, skipping >2012-11-08 04:15:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-5) Autorecovering hosts is disabled, skipping >2012-11-08 04:15:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-5) Autorecovering storage domains is disabled, skipping >2012-11-08 04:19:55,932 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-25) DbUserCacheManager::refreshAllUserData() - entered >2012-11-08 04:19:58,776 INFO [org.ovirt.engine.core.bll.LogoutUserCommand] (ajp-/127.0.0.1:8702-5) [490f1f74] Running command: LogoutUserCommand internal: false. >2012-11-08 04:20:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-33) Autorecovering hosts is disabled, skipping >2012-11-08 04:20:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-33) Autorecovering storage domains is disabled, skipping >2012-11-08 04:25:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-60) Autorecovering hosts is disabled, skipping >2012-11-08 04:25:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-60) Autorecovering storage domains is disabled, skipping >2012-11-08 04:30:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-87) Autorecovering hosts is disabled, skipping >2012-11-08 04:30:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-87) Autorecovering storage domains is disabled, skipping >2012-11-08 04:35:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-13) Autorecovering hosts is disabled, skipping >2012-11-08 04:35:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-13) Autorecovering storage domains is disabled, skipping >2012-11-08 04:40:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-38) Autorecovering hosts is disabled, skipping >2012-11-08 04:40:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-38) Autorecovering storage domains is disabled, skipping >2012-11-08 04:45:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-67) Autorecovering hosts is disabled, skipping >2012-11-08 04:45:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-67) Autorecovering storage domains is disabled, skipping >2012-11-08 04:50:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-94) Autorecovering hosts is disabled, skipping >2012-11-08 04:50:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-94) Autorecovering storage domains is disabled, skipping >2012-11-08 04:55:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-20) Autorecovering hosts is disabled, skipping >2012-11-08 04:55:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-20) Autorecovering storage domains is disabled, skipping >2012-11-08 05:00:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-47) Autorecovering hosts is disabled, skipping >2012-11-08 05:00:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-47) Autorecovering storage domains is disabled, skipping >2012-11-08 05:03:37,213 INFO [org.ovirt.engine.core.bll.LoginAdminUserCommand] (ajp-/127.0.0.1:8702-8) Checking if user admin@internal is an admin, result true >2012-11-08 05:03:37,217 INFO [org.ovirt.engine.core.bll.LoginAdminUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginAdminUserCommand internal: false. >2012-11-08 05:05:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-72) Autorecovering hosts is disabled, skipping >2012-11-08 05:05:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-72) Autorecovering storage domains is disabled, skipping >2012-11-08 05:08:00,308 INFO [org.ovirt.engine.core.bll.LoginAdminUserCommand] (ajp-/127.0.0.1:8702-12) Checking if user admin@internal is an admin, result true >2012-11-08 05:08:00,309 INFO [org.ovirt.engine.core.bll.LoginAdminUserCommand] (ajp-/127.0.0.1:8702-12) Running command: LoginAdminUserCommand internal: false. >2012-11-08 05:10:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-99) Autorecovering hosts is disabled, skipping >2012-11-08 05:10:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-99) Autorecovering storage domains is disabled, skipping >2012-11-08 05:15:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-25) Autorecovering hosts is disabled, skipping >2012-11-08 05:15:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-25) Autorecovering storage domains is disabled, skipping >2012-11-08 05:19:55,934 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-46) DbUserCacheManager::refreshAllUserData() - entered >2012-11-08 05:20:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-55) Autorecovering hosts is disabled, skipping >2012-11-08 05:20:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-55) Autorecovering storage domains is disabled, skipping >2012-11-08 05:22:11,275 INFO [org.ovirt.engine.core.bll.LoginAdminUserCommand] (ajp-/127.0.0.1:8702-10) Checking if user admin@internal is an admin, result true >2012-11-08 05:22:11,276 INFO [org.ovirt.engine.core.bll.LoginAdminUserCommand] (ajp-/127.0.0.1:8702-10) Running command: LoginAdminUserCommand internal: false. >2012-11-08 05:25:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-81) Autorecovering hosts is disabled, skipping >2012-11-08 05:25:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-81) Autorecovering storage domains is disabled, skipping >2012-11-08 05:30:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-10) Autorecovering hosts is disabled, skipping >2012-11-08 05:30:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-10) Autorecovering storage domains is disabled, skipping >2012-11-08 05:35:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-34) Autorecovering hosts is disabled, skipping >2012-11-08 05:35:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-34) Autorecovering storage domains is disabled, skipping >2012-11-08 05:39:34,899 INFO [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (ajp-/127.0.0.1:8702-7) SSH execute 10.70.36.15 'IDFILE=/etc/vdsm/vdsm.id; if [ -r "${IDFILE}" ]; then cat "${IDFILE}"; else UUID="$(dmidecode -s system-uuid 2> /dev/null | sed -e 's/.*Not.*//' )"; if [ -z "${UUID}" ]; then UUID="$(uuidgen 2> /dev/null)" && mkdir -p "$(dirname "${IDFILE}")" && echo "${UUID}" > "${IDFILE}" && chmod 0644 "${IDFILE}"; fi; [ -n "${UUID}" ] && echo "${UUID}"; fi' >2012-11-08 05:39:35,068 INFO [org.ovirt.engine.core.bll.AddVdsCommand] (ajp-/127.0.0.1:8702-7) [2bbd73f2] Running command: AddVdsCommand internal: false. Entities affected : ID: 6cea47a6-2976-11e2-ad99-52540035ed48 Type: VdsGroups >2012-11-08 05:39:35,087 INFO [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp-/127.0.0.1:8702-7) [3a3ee99f] Running command: AddVdsSpmIdCommand internal: true. Entities affected : ID: 96739be4-2990-11e2-8577-52540035ed48 Type: VDS >2012-11-08 05:39:35,098 INFO [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp-/127.0.0.1:8702-7) [3a3ee99f] Lock freed to object EngineLock [exclusiveLocks= key: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 value: REGISTER_VDS >, sharedLocks= ] >2012-11-08 05:39:35,099 ERROR [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp-/127.0.0.1:8702-7) [3a3ee99f] Cannot get vdsManager for vdsid=96739be4-2990-11e2-8577-52540035ed48 >2012-11-08 05:39:35,100 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (ajp-/127.0.0.1:8702-7) [3a3ee99f] START, RemoveVdsVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48), log id: 38688efd >2012-11-08 05:39:35,100 ERROR [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp-/127.0.0.1:8702-7) [3a3ee99f] Cannot get vdsManager for vdsid=96739be4-2990-11e2-8577-52540035ed48 >2012-11-08 05:39:35,100 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (ajp-/127.0.0.1:8702-7) [3a3ee99f] FINISH, RemoveVdsVDSCommand, log id: 38688efd >2012-11-08 05:39:35,100 ERROR [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp-/127.0.0.1:8702-7) [3a3ee99f] Cannot get vdsManager for vdsid=96739be4-2990-11e2-8577-52540035ed48 >2012-11-08 05:39:35,101 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-7) [3a3ee99f] START, AddVdsVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48), log id: 2c9b7b83 >2012-11-08 05:39:35,101 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-7) [3a3ee99f] AddVds - entered , starting logic to add VDS 96739be4-2990-11e2-8577-52540035ed48 >2012-11-08 05:39:35,102 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-7) [3a3ee99f] AddVds - VDS 96739be4-2990-11e2-8577-52540035ed48 was added, will try to add it to the resource manager >2012-11-08 05:39:35,103 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ajp-/127.0.0.1:8702-7) [3a3ee99f] Eneterd VdsManager:constructor >2012-11-08 05:39:35,105 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ajp-/127.0.0.1:8702-7) [3a3ee99f] vdsBroker(10.70.36.15,54,321) >2012-11-08 05:39:35,109 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp-/127.0.0.1:8702-7) [3a3ee99f] ResourceManager::AddVds - VDS 96739be4-2990-11e2-8577-52540035ed48 was added to the Resource Manager >2012-11-08 05:39:35,109 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-7) [3a3ee99f] FINISH, AddVdsVDSCommand, log id: 2c9b7b83 >2012-11-08 05:39:35,129 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-4-thread-50) [40d97ecc] Running command: InstallVdsCommand internal: true. Entities affected : ID: 96739be4-2990-11e2-8577-52540035ed48 Type: VDS >2012-11-08 05:39:35,130 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-4-thread-50) [40d97ecc] Before Installation pool-4-thread-50 >2012-11-08 05:39:35,130 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-50) [40d97ecc] Installation of 10.70.36.15. Executing installation stage. (Stage: Starting Host installation) >2012-11-08 05:39:35,130 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-50) [40d97ecc] Installation of 10.70.36.15. Executing installation stage. (Stage: Connecting to Host) >2012-11-08 05:39:35,194 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-50) [40d97ecc] Installation of 10.70.36.15. Received message: <BSTRAP component='RHEV_INSTALL' status='OK' message='Connected to Host 10.70.36.15 with SSH key fingerprint: cc:0f:a3:d7:15:b2:45:74:3a:6b:c6:2b:4b:f5:0d:0d'/>. FYI. (Stage: Connecting to Host) >2012-11-08 05:39:35,244 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-50) [40d97ecc] Installation of 10.70.36.15. Successfully connected to server ssh. (Stage: Connecting to Host) >2012-11-08 05:39:35,244 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-50) [40d97ecc] Installation of 10.70.36.15. Executing installation stage. (Stage: Get the unique vds id) >2012-11-08 05:39:35,244 INFO [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-4-thread-50) SSH execute 10.70.36.15 'IDFILE=/etc/vdsm/vdsm.id; if [ -r "${IDFILE}" ]; then cat "${IDFILE}"; else UUID="$(dmidecode -s system-uuid 2> /dev/null | sed -e 's/.*Not.*//' )"; if [ -z "${UUID}" ]; then UUID="$(uuidgen 2> /dev/null)" && mkdir -p "$(dirname "${IDFILE}")" && echo "${UUID}" > "${IDFILE}" && chmod 0644 "${IDFILE}"; fi; [ -n "${UUID}" ] && echo "${UUID}"; fi' >2012-11-08 05:39:35,360 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-24) Installation of 10.70.36.15. Received message: 00000000-0000-0000-0000-0025902F7254. FYI. (Stage: Get the unique vds id) >2012-11-08 05:39:35,362 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-24) Installation of 10.70.36.15. Assigning unique id 00000000-0000-0000-0000-0025902F7254 to Host. (Stage: Get the unique vds id) >2012-11-08 05:39:35,365 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-50) [40d97ecc] Installation of 10.70.36.15. Executing installation stage. (Stage: Upload Installation script to Host) >2012-11-08 05:39:35,365 INFO [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-4-thread-50) SSH execute 10.70.36.15 'cat > '/tmp/ovirt-id_rsa_efaa12af-7a46-44ac-bf86-cb9448fbca5e'' >2012-11-08 05:39:35,450 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-50) [40d97ecc] Installation of 10.70.36.15. Executing installation stage. (Stage: Running first installation script on Host) >2012-11-08 05:39:35,450 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-50) [40d97ecc] Installation of 10.70.36.15. Sending SSH Command umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar -C "${MYTMP}" --no-same-permissions -o -x && "${MYTMP}"/setup -c 'ssl=true;management_port=54321' -O 'lab.eng.blr.redhat.com' -t 2012-11-08T10:39:35 -S /tmp/ovirt-id_rsa_efaa12af-7a46-44ac-bf86-cb9448fbca5e -p 80 -b -B rhevm http://guido.lab.eng.blr.redhat.com:80/Components/vds/ http://guido.lab.eng.blr.redhat.com:80/Components/vds/ 10.70.36.15 efaa12af-7a46-44ac-bf86-cb9448fbca5e False < /var/cache/ovirt-engine/vdsm-bootstrap-2.tar. (Stage: Running first installation script on Host) >2012-11-08 05:39:35,451 INFO [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-4-thread-50) SSH execute 10.70.36.15 'umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar -C "${MYTMP}" --no-same-permissions -o -x && "${MYTMP}"/setup -c 'ssl=true;management_port=54321' -O 'lab.eng.blr.redhat.com' -t 2012-11-08T10:39:35 -S /tmp/ovirt-id_rsa_efaa12af-7a46-44ac-bf86-cb9448fbca5e -p 80 -b -B rhevm http://guido.lab.eng.blr.redhat.com:80/Components/vds/ http://guido.lab.eng.blr.redhat.com:80/Components/vds/ 10.70.36.15 efaa12af-7a46-44ac-bf86-cb9448fbca5e False' >2012-11-08 05:39:35,557 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-24) Installation of 10.70.36.15. Received message: <BSTRAP component='INSTALLER' status='OK' message='Test platform succeeded'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:39:36,670 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-24) Installation of 10.70.36.15. Received message: <BSTRAP component='SET_SYSTEM_TIME' status='OK' message='setSystemTime ended successfully'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:39:37,110 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-38) Initializing Host: rhs-gp-srv11.lab.eng.blr.redhat.com >2012-11-08 05:39:38,462 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-24) Installation of 10.70.36.15. Received message: <BSTRAP component='RHN_REGISTRATION' status='OK' message='Host properly registered with RHN/Satellite.'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:39:38,609 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-24) Installation of 10.70.36.15. Received message: <BSTRAP component='VDSM_MAJOR_VER' status='OK' message='Available VDSM matches requirements'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:39:38,615 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-24) Installation of 10.70.36.15. Received message: <BSTRAP component='VT_SVM' status='OK' processor='Intel' message='Server supports virtualization'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:39:38,621 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-24) Installation of 10.70.36.15. Received message: <BSTRAP component='OS' status='OK' type='RHEL6' message='Supported platform version'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:39:38,630 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-24) Installation of 10.70.36.15. Received message: <BSTRAP component='KERNEL' status='OK' version='279' message='Supported kernel version: 279'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:39:38,694 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-24) Installation of 10.70.36.15. Received message: <BSTRAP component='REQ PACKAGES' status='WARN' result='SDL.x86_64' message='package SDL.x86_64 is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:39:38,710 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-24) Installation of 10.70.36.15. Received message: <BSTRAP component='REQ PACKAGES' status='OK' result='bridge-utils.x86_64' message='bridge-utils-1.2-9.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:39:38,723 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-24) Installation of 10.70.36.15. Received message: <BSTRAP component='REQ PACKAGES' status='WARN' result='mesa-libGLU.x86_64' message='package mesa-libGLU.x86_64 is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:39:38,737 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-24) Installation of 10.70.36.15. Received message: <BSTRAP component='REQ PACKAGES' status='OK' result='openssl.x86_64' message='openssl-1.0.0-25.el6_3.1.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:39:38,752 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-24) Installation of 10.70.36.15. Received message: <BSTRAP component='REQ PACKAGES' status='OK' result='m2crypto.x86_64' message='m2crypto-0.20.2-9.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:39:38,766 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-24) Installation of 10.70.36.15. Received message: <BSTRAP component='REQ PACKAGES' status='OK' result='rsync.x86_64' message='rsync-3.0.6-9.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:39:38,780 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-24) Installation of 10.70.36.15. Received message: <BSTRAP component='VDS PACKAGES' status='WARN' result='qemu-kvm' message='package qemu-kvm is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:39:38,793 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-24) Installation of 10.70.36.15. Received message: <BSTRAP component='VDS PACKAGES' status='WARN' result='qemu-kvm-tools' message='package qemu-kvm-tools is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:39:38,807 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-24) Installation of 10.70.36.15. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='(~vdsm~, ~vdsm-cli~)' message='vdsm-4.9.6-41.0.el6_3.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:39:38,821 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-24) Installation of 10.70.36.15. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='libjpeg' message='libjpeg-6b-46.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:39:38,834 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-24) Installation of 10.70.36.15. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='spice-server' message='spice-server-0.10.1-10.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:39:38,848 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-24) Installation of 10.70.36.15. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='pixman' message='pixman-0.18.4-1.el6_0.1.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:39:38,862 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-24) Installation of 10.70.36.15. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='seabios' message='seabios-0.6.1.2-19.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:39:38,875 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-24) Installation of 10.70.36.15. Received message: <BSTRAP component='VDS PACKAGES' status='WARN' result='qemu-img' message='package qemu-img is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:39:38,889 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-24) Installation of 10.70.36.15. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='fence-agents' message='fence-agents-3.1.5-17.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:39:38,904 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-24) Installation of 10.70.36.15. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='libselinux-python' message='libselinux-python-2.0.94-5.3.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:39:38,917 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-24) Installation of 10.70.36.15. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='sanlock' message='sanlock-2.3-4.el6_3.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:39:38,931 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-24) Installation of 10.70.36.15. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='sanlock-python' message='sanlock-python-2.3-4.el6_3.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:40:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-73) Autorecovering hosts is disabled, skipping >2012-11-08 05:40:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-73) Autorecovering storage domains is disabled, skipping >2012-11-08 05:40:58,292 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-24) Installation of 10.70.36.15. Received message: <BSTRAP component='REQ PACKAGES' status='OK' result='mesa-libGLU.x86_64' message='mesa-libGLU-7.11-5.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:41:14,018 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-24) Installation of 10.70.36.15. Received message: <BSTRAP component='REQ PACKAGES' status='OK' result='SDL.x86_64' message='SDL-1.2.14-3.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:41:16,720 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-24) Installation of 10.70.36.15. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='(~vdsm~, ~vdsm-cli~)' message='vdsm-4.9.6-41.0.el6_3.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:41:19,294 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-24) Installation of 10.70.36.15. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='libjpeg' message='libjpeg-6b-46.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:41:22,007 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-24) Installation of 10.70.36.15. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='spice-server' message='spice-server-0.10.1-10.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:41:24,800 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-24) Installation of 10.70.36.15. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='pixman' message='pixman-0.18.4-1.el6_0.1.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:41:27,451 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-24) Installation of 10.70.36.15. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='seabios' message='seabios-0.6.1.2-19.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:41:29,737 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-24) Installation of 10.70.36.15. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='fence-agents' message='fence-agents-3.1.5-17.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:41:32,155 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-24) Installation of 10.70.36.15. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='libselinux-python' message='libselinux-python-2.0.94-5.3.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:41:35,119 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-24) Installation of 10.70.36.15. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='sanlock' message='sanlock-2.3-4.el6_3.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:41:37,818 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-24) Installation of 10.70.36.15. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='sanlock-python' message='sanlock-python-2.3-4.el6_3.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:41:40,296 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-24) Installation of 10.70.36.15. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='qemu-img' message='package qemu-img is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:41:53,447 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-24) Installation of 10.70.36.15. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='qemu-kvm-tools' message='package qemu-kvm-tools is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:41:55,650 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-24) Installation of 10.70.36.15. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='qemu-kvm' message='package qemu-kvm is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:41:55,654 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-24) Installation of 10.70.36.15. Received message: <BSTRAP component='CreateConf' status='OK' message='Basic configuration set'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:41:55,665 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-24) Installation of 10.70.36.15. Received message: <BSTRAP component='CheckLocalHostname' status='OK' message='Local hostname is correct.'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:42:06,257 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-24) Installation of 10.70.36.15. Received message: <BSTRAP component='SetNetworking' status='FAIL' message='addNetwork error trying to add management bridge'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:42:06,263 ERROR [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-24) Installation of 10.70.36.15. Received message: <BSTRAP component='RHEV_INSTALL' status='FAIL'/>. Error occured. (Stage: Running first installation script on Host) >2012-11-08 05:42:06,296 ERROR [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-50) [40d97ecc] Installation of 10.70.36.15. Error: Repository JBOSS is listed more than once in the configuration >Repository qa-latest-5 is listed more than once in the configuration >Repository rhel-source is listed more than once in the configuration >Repository rhel-source-beta is listed more than once in the configuration >Repository rhs-client is listed more than once in the configuration >Repository JBOSS is listed more than once in the configuration >Repository qa-latest-5 is listed more than once in the configuration >Repository rhel-source is listed more than once in the configuration >Repository rhel-source-beta is listed more than once in the configuration >Repository rhs-client is listed more than once in the configuration >. (Stage: Running first installation script on Host) >2012-11-08 05:42:06,296 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-50) [40d97ecc] VDS message: Repository JBOSS is listed more than once in the configuration >2012-11-08 05:42:06,296 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-50) [40d97ecc] VDS message: Repository qa-latest-5 is listed more than once in the configuration >2012-11-08 05:42:06,296 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-50) [40d97ecc] VDS message: Repository rhel-source is listed more than once in the configuration >2012-11-08 05:42:06,296 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-50) [40d97ecc] VDS message: Repository rhel-source-beta is listed more than once in the configuration >2012-11-08 05:42:06,297 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-50) [40d97ecc] VDS message: Repository rhs-client is listed more than once in the configuration >2012-11-08 05:42:06,297 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-50) [40d97ecc] VDS message: Repository JBOSS is listed more than once in the configuration >2012-11-08 05:42:06,297 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-50) [40d97ecc] VDS message: Repository qa-latest-5 is listed more than once in the configuration >2012-11-08 05:42:06,297 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-50) [40d97ecc] VDS message: Repository rhel-source is listed more than once in the configuration >2012-11-08 05:42:06,297 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-50) [40d97ecc] VDS message: Repository rhel-source-beta is listed more than once in the configuration >2012-11-08 05:42:06,297 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-50) [40d97ecc] VDS message: Repository rhs-client is listed more than once in the configuration >2012-11-08 05:42:06,297 ERROR [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-4-thread-50) SSH error running command 10.70.36.15:'umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar -C "${MYTMP}" --no-same-permissions -o -x && "${MYTMP}"/setup -c 'ssl=true;management_port=54321' -O 'lab.eng.blr.redhat.com' -t 2012-11-08T10:39:35 -S /tmp/ovirt-id_rsa_efaa12af-7a46-44ac-bf86-cb9448fbca5e -p 80 -b -B rhevm http://guido.lab.eng.blr.redhat.com:80/Components/vds/ http://guido.lab.eng.blr.redhat.com:80/Components/vds/ 10.70.36.15 efaa12af-7a46-44ac-bf86-cb9448fbca5e False': java.io.IOException: Command returned failure code 1 during SSH session 10.70.36.15:22' 'umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar -C "${MYTMP}" --no-same-permissions -o -x && "${MYTMP}"/setup -c 'ssl=true;management_port=54321' -O 'lab.eng.blr.redhat.com' -t 2012-11-08T10:39:35 -S /tmp/ovirt-id_rsa_efaa12af-7a46-44ac-bf86-cb9448fbca5e -p 80 -b -B rhevm http://guido.lab.eng.blr.redhat.com:80/Components/vds/ http://guido.lab.eng.blr.redhat.com:80/Components/vds/ 10.70.36.15 efaa12af-7a46-44ac-bf86-cb9448fbca5e False' > at org.ovirt.engine.core.utils.ssh.SSHClient.executeCommand(SSHClient.java:442) [engine-utils.jar:] > at org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH.executeCommand(VdsInstallerSSH.java:387) [engine-utils.jar:] > at org.ovirt.engine.core.bll.VdsInstaller.runBootstrapCommand(VdsInstaller.java:217) [engine-bll.jar:] > at org.ovirt.engine.core.bll.VdsInstaller.RunStage(VdsInstaller.java:341) [engine-bll.jar:] > at org.ovirt.engine.core.bll.VdsInstaller.Install(VdsInstaller.java:280) [engine-bll.jar:] > at org.ovirt.engine.core.bll.InstallVdsCommand.executeCommand(InstallVdsCommand.java:110) [engine-bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:825) [engine-bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:916) [engine-bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1300) [engine-bll.jar:] > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:168) [engine-utils.jar:] > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:107) [engine-utils.jar:] > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:931) [engine-bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:285) [engine-bll.jar:] > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:336) [engine-bll.jar:] > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:593) [engine-bll.jar:] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_09-icedtea] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_09-icedtea] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_09-icedtea] > at java.lang.reflect.Method.invoke(Method.java:601) [rt.jar:1.7.0_09-icedtea] > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) [jboss-as-ee.jar:7.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:36) [jboss-as-ee.jar:7.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) [jboss-as-ejb3.jar:7.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) [jboss-as-ee.jar:7.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:53) [jboss-as-ejb3.jar:7.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:210) [jboss-as-ejb3.jar:7.1.2.Final-redhat-1] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:362) [jboss-as-ejb3.jar:7.1.2.Final-redhat-1] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:193) [jboss-as-ejb3.jar:7.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [jboss-as-ejb3.jar:7.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:42) [jboss-as-ejb3.jar:7.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) [jboss-as-ejb3.jar:7.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) [jboss-as-ee.jar:7.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) [jboss-as-ee.jar:7.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) [jboss-as-ee.jar:7.1.2.Final-redhat-1] > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:176) [jboss-as-ee.jar:7.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) [jboss-as-ee.jar:7.1.2.Final-redhat-1] > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) [engine-bll.jar:] > at org.ovirt.engine.core.bll.AddVdsCommand$3.run(AddVdsCommand.java:156) [engine-bll.jar:] > at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalWrapperRunnable.run(ThreadPoolUtil.java:64) [engine-utils.jar:] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_09-icedtea] > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) [rt.jar:1.7.0_09-icedtea] > at java.util.concurrent.FutureTask.run(FutureTask.java:166) [rt.jar:1.7.0_09-icedtea] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [rt.jar:1.7.0_09-icedtea] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [rt.jar:1.7.0_09-icedtea] > at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_09-icedtea] > >2012-11-08 05:42:06,299 ERROR [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-50) [40d97ecc] Installation of 10.70.36.15 has failed. Failure details: SSH command failed while executing at host '10.70.36.15', refer to logs for further information. (Stage: Running first installation script on Host) >2012-11-08 05:42:06,299 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-50) [40d97ecc] VDS message: SSH command failed while executing at host '10.70.36.15', refer to logs for further information >2012-11-08 05:42:06,299 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-50) [40d97ecc] Script ended, result is {1} >2012-11-08 05:42:06,299 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-4-thread-50) [40d97ecc] After Installation pool-4-thread-50 >2012-11-08 05:42:06,303 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-50) [40d97ecc] START, SetVdsStatusVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, status=InstallFailed, nonOperationalReason=NONE), log id: 20552df8 >2012-11-08 05:42:06,316 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-50) [40d97ecc] FINISH, SetVdsStatusVDSCommand, log id: 20552df8 >2012-11-08 05:45:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-49) Autorecovering hosts is disabled, skipping >2012-11-08 05:45:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-49) Autorecovering storage domains is disabled, skipping >2012-11-08 05:46:06,519 INFO [org.ovirt.engine.core.bll.MaintananceNumberOfVdssCommand] (pool-4-thread-50) [23b715ce] Running command: MaintananceNumberOfVdssCommand internal: false. Entities affected : ID: 96739be4-2990-11e2-8577-52540035ed48 Type: VDS >2012-11-08 05:46:06,520 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-50) [23b715ce] START, SetVdsStatusVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, status=PreparingForMaintenance, nonOperationalReason=NONE), log id: 4fe7ec0e >2012-11-08 05:46:06,525 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-50) [23b715ce] FINISH, SetVdsStatusVDSCommand, log id: 4fe7ec0e >2012-11-08 05:46:06,555 INFO [org.ovirt.engine.core.bll.MaintananceVdsCommand] (pool-4-thread-50) [23b715ce] Running command: MaintananceVdsCommand internal: true. Entities affected : ID: 96739be4-2990-11e2-8577-52540035ed48 Type: VDS >2012-11-08 05:46:07,487 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-31) vds::refreshVdsStats Failed getVdsStats, vds = 96739be4-2990-11e2-8577-52540035ed48 : rhs-gp-srv11.lab.eng.blr.redhat.com, error = VDSNetworkException: VDSNetworkException: >2012-11-08 05:46:07,487 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-31) ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = 96739be4-2990-11e2-8577-52540035ed48 : rhs-gp-srv11.lab.eng.blr.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-11-08 05:46:09,517 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-36) vds::refreshVdsStats Failed getVdsStats, vds = 96739be4-2990-11e2-8577-52540035ed48 : rhs-gp-srv11.lab.eng.blr.redhat.com, error = VDSNetworkException: VDSNetworkException: >2012-11-08 05:46:09,518 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-36) ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = 96739be4-2990-11e2-8577-52540035ed48 : rhs-gp-srv11.lab.eng.blr.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-11-08 05:46:11,529 ERROR [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-37) VDS::handleNetworkException Server failed to respond, vds_id = 96739be4-2990-11e2-8577-52540035ed48, vds_name = rhs-gp-srv11.lab.eng.blr.redhat.com, error = VDSNetworkException: >2012-11-08 05:46:11,540 INFO [org.ovirt.engine.core.bll.VdsEventListener] (pool-4-thread-50) ResourceManager::vdsNotResponding entered for Host 96739be4-2990-11e2-8577-52540035ed48, 10.70.36.15 >2012-11-08 05:46:11,543 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-37) vds::refreshVdsStats Failed getVdsStats, vds = 96739be4-2990-11e2-8577-52540035ed48 : rhs-gp-srv11.lab.eng.blr.redhat.com, error = VDSNetworkException: VDSNetworkException: >2012-11-08 05:46:11,556 WARN [org.ovirt.engine.core.bll.VdsNotRespondingTreatmentCommand] (pool-4-thread-50) [75a09ba7] CanDoAction of action VdsNotRespondingTreatment failed. Reasons:VDS_FENCING_DISABLED >2012-11-08 05:47:27,956 INFO [org.ovirt.engine.core.bll.LoginAdminUserCommand] (ajp-/127.0.0.1:8702-7) Checking if user admin@internal is an admin, result true >2012-11-08 05:47:27,956 INFO [org.ovirt.engine.core.bll.LoginAdminUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginAdminUserCommand internal: false. >2012-11-08 05:48:33,659 INFO [org.ovirt.engine.core.bll.MaintananceNumberOfVdssCommand] (pool-4-thread-50) [2f2efce0] Running command: MaintananceNumberOfVdssCommand internal: false. Entities affected : ID: 96739be4-2990-11e2-8577-52540035ed48 Type: VDS >2012-11-08 05:48:33,669 INFO [org.ovirt.engine.core.bll.MaintananceVdsCommand] (pool-4-thread-50) [2f2efce0] Running command: MaintananceVdsCommand internal: true. Entities affected : ID: 96739be4-2990-11e2-8577-52540035ed48 Type: VDS >2012-11-08 05:48:33,671 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-50) [2f2efce0] START, SetVdsStatusVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, status=Maintenance, nonOperationalReason=NONE), log id: 296ddb2c >2012-11-08 05:48:33,673 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-50) [2f2efce0] FINISH, SetVdsStatusVDSCommand, log id: 296ddb2c >2012-11-08 05:48:38,003 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (ajp-/127.0.0.1:8702-11) Lock Acquired to object EngineLock [exclusiveLocks= key: 96739be4-2990-11e2-8577-52540035ed48 value: VDS >, sharedLocks= ] >2012-11-08 05:48:38,018 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (pool-4-thread-50) [64337e90] Running command: ActivateVdsCommand internal: false. Entities affected : ID: 96739be4-2990-11e2-8577-52540035ed48 Type: VDS >2012-11-08 05:48:38,024 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-50) [64337e90] START, SetVdsStatusVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, status=Unassigned, nonOperationalReason=NONE), log id: 3c55e30e >2012-11-08 05:48:38,026 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-50) [64337e90] FINISH, SetVdsStatusVDSCommand, log id: 3c55e30e >2012-11-08 05:48:38,027 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (pool-4-thread-50) [64337e90] START, ActivateVdsVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48), log id: 32826d5 >2012-11-08 05:48:38,036 ERROR [org.ovirt.engine.core.vdsbroker.VdsManager] (pool-4-thread-50) [64337e90] VDS::handleNetworkException Server failed to respond, vds_id = 96739be4-2990-11e2-8577-52540035ed48, vds_name = rhs-gp-srv11.lab.eng.blr.redhat.com, error = VDSNetworkException: >2012-11-08 05:48:38,049 INFO [org.ovirt.engine.core.bll.VdsEventListener] (pool-4-thread-49) [64337e90] ResourceManager::vdsNotResponding entered for Host 96739be4-2990-11e2-8577-52540035ed48, 10.70.36.15 >2012-11-08 05:48:38,051 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (pool-4-thread-50) [64337e90] ResourceManager::activateVds - failed to get VDS = 96739be4-2990-11e2-8577-52540035ed48 capabilities with error: VDSNetworkException: . >2012-11-08 05:48:38,051 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (pool-4-thread-50) [64337e90] ResourceManager::activateVds - failed to activate VDS = 96739be4-2990-11e2-8577-52540035ed48 >2012-11-08 05:48:38,055 WARN [org.ovirt.engine.core.bll.VdsNotRespondingTreatmentCommand] (pool-4-thread-49) [77e79d07] CanDoAction of action VdsNotRespondingTreatment failed. Reasons:VDS_FENCING_DISABLED >2012-11-08 05:48:38,057 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (pool-4-thread-50) [508d7063] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 96739be4-2990-11e2-8577-52540035ed48 Type: VDS >2012-11-08 05:48:38,063 WARN [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (pool-4-thread-50) [35574f2a] CanDoAction of action HandleVdsVersion failed. Reasons:VDS_CANNOT_CHECK_VERSION_HOST_NON_RESPONSIVE >2012-11-08 05:48:38,064 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (pool-4-thread-50) [35574f2a] FINISH, ActivateVdsVDSCommand, log id: 32826d5 >2012-11-08 05:48:38,077 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (pool-4-thread-50) [35574f2a] Lock freed to object EngineLock [exclusiveLocks= key: 96739be4-2990-11e2-8577-52540035ed48 value: VDS >, sharedLocks= ] >2012-11-08 05:49:30,559 INFO [org.ovirt.engine.core.bll.MaintananceNumberOfVdssCommand] (pool-4-thread-50) [53e4f79b] Running command: MaintananceNumberOfVdssCommand internal: false. Entities affected : ID: 96739be4-2990-11e2-8577-52540035ed48 Type: VDS >2012-11-08 05:49:30,577 INFO [org.ovirt.engine.core.bll.MaintananceVdsCommand] (pool-4-thread-50) [53e4f79b] Running command: MaintananceVdsCommand internal: true. Entities affected : ID: 96739be4-2990-11e2-8577-52540035ed48 Type: VDS >2012-11-08 05:49:30,581 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-50) [53e4f79b] START, SetVdsStatusVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, status=Maintenance, nonOperationalReason=NONE), log id: 39080d66 >2012-11-08 05:49:30,585 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-50) [53e4f79b] FINISH, SetVdsStatusVDSCommand, log id: 39080d66 >2012-11-08 05:49:41,837 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp-/127.0.0.1:8702-9) [6de09cae] Running command: UpdateVdsCommand internal: false. Entities affected : ID: 96739be4-2990-11e2-8577-52540035ed48 Type: VDS >2012-11-08 05:49:41,857 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp-/127.0.0.1:8702-9) [6de09cae] START, SetVdsStatusVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, status=Installing, nonOperationalReason=NONE), log id: 13016c84 >2012-11-08 05:49:41,859 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp-/127.0.0.1:8702-9) [6de09cae] FINISH, SetVdsStatusVDSCommand, log id: 13016c84 >2012-11-08 05:49:41,861 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-4-thread-50) [22279fd7] Running command: InstallVdsCommand internal: true. Entities affected : ID: 96739be4-2990-11e2-8577-52540035ed48 Type: VDS >2012-11-08 05:49:41,862 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-4-thread-50) [22279fd7] Before Installation pool-4-thread-50 >2012-11-08 05:49:41,862 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-50) [22279fd7] Installation of 10.70.36.15. Executing installation stage. (Stage: Starting Host installation) >2012-11-08 05:49:41,862 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-50) [22279fd7] Installation of 10.70.36.15. Executing installation stage. (Stage: Connecting to Host) >2012-11-08 05:49:41,928 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-50) [22279fd7] Installation of 10.70.36.15. Received message: <BSTRAP component='RHEV_INSTALL' status='OK' message='Connected to Host 10.70.36.15 with SSH key fingerprint: cc:0f:a3:d7:15:b2:45:74:3a:6b:c6:2b:4b:f5:0d:0d'/>. FYI. (Stage: Connecting to Host) >2012-11-08 05:49:41,958 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-50) [22279fd7] Installation of 10.70.36.15. Successfully connected to server ssh. (Stage: Connecting to Host) >2012-11-08 05:49:41,958 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-50) [22279fd7] Installation of 10.70.36.15. Executing installation stage. (Stage: Get the unique vds id) >2012-11-08 05:49:41,958 INFO [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-4-thread-50) SSH execute 10.70.36.15 'IDFILE=/etc/vdsm/vdsm.id; if [ -r "${IDFILE}" ]; then cat "${IDFILE}"; else UUID="$(dmidecode -s system-uuid 2> /dev/null | sed -e 's/.*Not.*//' )"; if [ -z "${UUID}" ]; then UUID="$(uuidgen 2> /dev/null)" && mkdir -p "$(dirname "${IDFILE}")" && echo "${UUID}" > "${IDFILE}" && chmod 0644 "${IDFILE}"; fi; [ -n "${UUID}" ] && echo "${UUID}"; fi' >2012-11-08 05:49:42,082 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-25) Installation of 10.70.36.15. Received message: 00000000-0000-0000-0000-0025902F7254. FYI. (Stage: Get the unique vds id) >2012-11-08 05:49:42,084 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-25) Installation of 10.70.36.15. Assigning unique id 00000000-0000-0000-0000-0025902F7254 to Host. (Stage: Get the unique vds id) >2012-11-08 05:49:42,087 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-50) [22279fd7] Installation of 10.70.36.15. Executing installation stage. (Stage: Upload Installation script to Host) >2012-11-08 05:49:42,087 INFO [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-4-thread-50) SSH execute 10.70.36.15 'cat > '/tmp/ovirt-id_rsa_c05b1fdf-4784-46e5-8384-fb5ba869c49f'' >2012-11-08 05:49:42,176 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-50) [22279fd7] Installation of 10.70.36.15. Executing installation stage. (Stage: Running first installation script on Host) >2012-11-08 05:49:42,176 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-50) [22279fd7] Installation of 10.70.36.15. Sending SSH Command umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar -C "${MYTMP}" --no-same-permissions -o -x && "${MYTMP}"/setup -c 'ssl=true;management_port=54321' -O 'lab.eng.blr.redhat.com' -t 2012-11-08T10:49:42 -S /tmp/ovirt-id_rsa_c05b1fdf-4784-46e5-8384-fb5ba869c49f -p 80 -b -B rhevm http://guido.lab.eng.blr.redhat.com:80/Components/vds/ http://guido.lab.eng.blr.redhat.com:80/Components/vds/ 10.70.36.15 c05b1fdf-4784-46e5-8384-fb5ba869c49f False < /var/cache/ovirt-engine/vdsm-bootstrap-2.tar. (Stage: Running first installation script on Host) >2012-11-08 05:49:42,177 INFO [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-4-thread-50) SSH execute 10.70.36.15 'umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar -C "${MYTMP}" --no-same-permissions -o -x && "${MYTMP}"/setup -c 'ssl=true;management_port=54321' -O 'lab.eng.blr.redhat.com' -t 2012-11-08T10:49:42 -S /tmp/ovirt-id_rsa_c05b1fdf-4784-46e5-8384-fb5ba869c49f -p 80 -b -B rhevm http://guido.lab.eng.blr.redhat.com:80/Components/vds/ http://guido.lab.eng.blr.redhat.com:80/Components/vds/ 10.70.36.15 c05b1fdf-4784-46e5-8384-fb5ba869c49f False' >2012-11-08 05:49:42,270 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-25) Installation of 10.70.36.15. Received message: <BSTRAP component='INSTALLER' status='OK' message='Test platform succeeded'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:49:44,358 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-25) Installation of 10.70.36.15. Received message: <BSTRAP component='SET_SYSTEM_TIME' status='OK' message='setSystemTime ended successfully'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:49:46,130 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-25) Installation of 10.70.36.15. Received message: <BSTRAP component='RHN_REGISTRATION' status='OK' message='Host properly registered with RHN/Satellite.'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:49:46,276 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-25) Installation of 10.70.36.15. Received message: <BSTRAP component='VDSM_MAJOR_VER' status='OK' message='Available VDSM matches requirements'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:49:46,289 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-25) Installation of 10.70.36.15. Received message: <BSTRAP component='VT_SVM' status='OK' processor='Intel' message='Server supports virtualization'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:49:46,293 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-25) Installation of 10.70.36.15. Received message: <BSTRAP component='OS' status='OK' type='RHEL6' message='Supported platform version'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:49:46,301 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-25) Installation of 10.70.36.15. Received message: <BSTRAP component='KERNEL' status='OK' version='279' message='Supported kernel version: 279'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:49:46,339 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-25) Installation of 10.70.36.15. Received message: <BSTRAP component='REQ PACKAGES' status='OK' result='SDL.x86_64' message='SDL-1.2.14-3.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:49:46,353 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-25) Installation of 10.70.36.15. Received message: <BSTRAP component='REQ PACKAGES' status='OK' result='bridge-utils.x86_64' message='bridge-utils-1.2-9.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:49:46,368 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-25) Installation of 10.70.36.15. Received message: <BSTRAP component='REQ PACKAGES' status='OK' result='mesa-libGLU.x86_64' message='mesa-libGLU-7.11-5.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:49:46,382 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-25) Installation of 10.70.36.15. Received message: <BSTRAP component='REQ PACKAGES' status='OK' result='openssl.x86_64' message='openssl-1.0.0-25.el6_3.1.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:49:46,397 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-25) Installation of 10.70.36.15. Received message: <BSTRAP component='REQ PACKAGES' status='OK' result='m2crypto.x86_64' message='m2crypto-0.20.2-9.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:49:46,411 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-25) Installation of 10.70.36.15. Received message: <BSTRAP component='REQ PACKAGES' status='OK' result='rsync.x86_64' message='rsync-3.0.6-9.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:49:46,425 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-25) Installation of 10.70.36.15. Received message: <BSTRAP component='VDS PACKAGES' status='WARN' result='qemu-kvm' message='package qemu-kvm is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:49:46,438 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-25) Installation of 10.70.36.15. Received message: <BSTRAP component='VDS PACKAGES' status='WARN' result='qemu-kvm-tools' message='package qemu-kvm-tools is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:49:46,452 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-25) Installation of 10.70.36.15. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='(~vdsm~, ~vdsm-cli~)' message='vdsm-4.9.6-41.0.el6_3.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:49:46,465 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-25) Installation of 10.70.36.15. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='libjpeg' message='libjpeg-6b-46.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:49:46,485 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-25) Installation of 10.70.36.15. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='spice-server' message='spice-server-0.10.1-10.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:49:46,493 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-25) Installation of 10.70.36.15. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='pixman' message='pixman-0.18.4-1.el6_0.1.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:49:46,507 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-25) Installation of 10.70.36.15. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='seabios' message='seabios-0.6.1.2-19.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:49:46,520 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-25) Installation of 10.70.36.15. Received message: <BSTRAP component='VDS PACKAGES' status='WARN' result='qemu-img' message='package qemu-img is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:49:46,534 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-25) Installation of 10.70.36.15. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='fence-agents' message='fence-agents-3.1.5-17.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:49:46,549 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-25) Installation of 10.70.36.15. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='libselinux-python' message='libselinux-python-2.0.94-5.3.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:49:46,562 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-25) Installation of 10.70.36.15. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='sanlock' message='sanlock-2.3-4.el6_3.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:49:46,575 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-25) Installation of 10.70.36.15. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='sanlock-python' message='sanlock-python-2.3-4.el6_3.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:50:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-25) Autorecovering hosts is disabled, skipping >2012-11-08 05:50:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-25) Autorecovering storage domains is disabled, skipping >2012-11-08 05:51:33,250 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-25) Installation of 10.70.36.15. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='(~vdsm~, ~vdsm-cli~)' message='vdsm-4.9.6-41.0.el6_3.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:51:35,752 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-25) Installation of 10.70.36.15. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='libjpeg' message='libjpeg-6b-46.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:51:37,991 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-25) Installation of 10.70.36.15. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='spice-server' message='spice-server-0.10.1-10.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:51:40,184 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-25) Installation of 10.70.36.15. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='pixman' message='pixman-0.18.4-1.el6_0.1.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:51:42,464 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-25) Installation of 10.70.36.15. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='seabios' message='seabios-0.6.1.2-19.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:51:44,689 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-25) Installation of 10.70.36.15. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='fence-agents' message='fence-agents-3.1.5-17.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:51:46,921 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-25) Installation of 10.70.36.15. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='libselinux-python' message='libselinux-python-2.0.94-5.3.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:51:49,110 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-25) Installation of 10.70.36.15. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='sanlock' message='sanlock-2.3-4.el6_3.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:51:51,345 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-25) Installation of 10.70.36.15. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='sanlock-python' message='sanlock-python-2.3-4.el6_3.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:51:53,576 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-25) Installation of 10.70.36.15. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='qemu-img' message='package qemu-img is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:51:55,810 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-25) Installation of 10.70.36.15. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='qemu-kvm-tools' message='package qemu-kvm-tools is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:51:58,038 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-25) Installation of 10.70.36.15. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='qemu-kvm' message='package qemu-kvm is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:51:58,044 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-25) Installation of 10.70.36.15. Received message: <BSTRAP component='CreateConf' status='OK' message='Basic configuration found, skipping this step'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:51:58,060 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-25) Installation of 10.70.36.15. Received message: <BSTRAP component='CheckLocalHostname' status='OK' message='Local hostname is correct.'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:52:08,459 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-25) Installation of 10.70.36.15. Received message: <BSTRAP component='SetNetworking' status='FAIL' message='addNetwork error trying to add management bridge'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:52:08,465 ERROR [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-25) Installation of 10.70.36.15. Received message: <BSTRAP component='RHEV_INSTALL' status='FAIL'/>. Error occured. (Stage: Running first installation script on Host) >2012-11-08 05:52:08,494 ERROR [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-50) [22279fd7] Installation of 10.70.36.15. Error: Repository JBOSS is listed more than once in the configuration >Repository qa-latest-5 is listed more than once in the configuration >Repository rhel-source is listed more than once in the configuration >Repository rhel-source-beta is listed more than once in the configuration >Repository rhs-client is listed more than once in the configuration >Repository JBOSS is listed more than once in the configuration >Repository qa-latest-5 is listed more than once in the configuration >Repository rhel-source is listed more than once in the configuration >Repository rhel-source-beta is listed more than once in the configuration >Repository rhs-client is listed more than once in the configuration >. (Stage: Running first installation script on Host) >2012-11-08 05:52:08,494 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-50) [22279fd7] VDS message: Repository JBOSS is listed more than once in the configuration >2012-11-08 05:52:08,494 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-50) [22279fd7] VDS message: Repository qa-latest-5 is listed more than once in the configuration >2012-11-08 05:52:08,494 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-50) [22279fd7] VDS message: Repository rhel-source is listed more than once in the configuration >2012-11-08 05:52:08,494 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-50) [22279fd7] VDS message: Repository rhel-source-beta is listed more than once in the configuration >2012-11-08 05:52:08,494 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-50) [22279fd7] VDS message: Repository rhs-client is listed more than once in the configuration >2012-11-08 05:52:08,494 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-50) [22279fd7] VDS message: Repository JBOSS is listed more than once in the configuration >2012-11-08 05:52:08,494 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-50) [22279fd7] VDS message: Repository qa-latest-5 is listed more than once in the configuration >2012-11-08 05:52:08,494 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-50) [22279fd7] VDS message: Repository rhel-source is listed more than once in the configuration >2012-11-08 05:52:08,494 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-50) [22279fd7] VDS message: Repository rhel-source-beta is listed more than once in the configuration >2012-11-08 05:52:08,494 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-50) [22279fd7] VDS message: Repository rhs-client is listed more than once in the configuration >2012-11-08 05:52:08,494 ERROR [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-4-thread-50) SSH error running command 10.70.36.15:'umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar -C "${MYTMP}" --no-same-permissions -o -x && "${MYTMP}"/setup -c 'ssl=true;management_port=54321' -O 'lab.eng.blr.redhat.com' -t 2012-11-08T10:49:42 -S /tmp/ovirt-id_rsa_c05b1fdf-4784-46e5-8384-fb5ba869c49f -p 80 -b -B rhevm http://guido.lab.eng.blr.redhat.com:80/Components/vds/ http://guido.lab.eng.blr.redhat.com:80/Components/vds/ 10.70.36.15 c05b1fdf-4784-46e5-8384-fb5ba869c49f False': java.io.IOException: Command returned failure code 1 during SSH session 10.70.36.15:22' 'umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar -C "${MYTMP}" --no-same-permissions -o -x && "${MYTMP}"/setup -c 'ssl=true;management_port=54321' -O 'lab.eng.blr.redhat.com' -t 2012-11-08T10:49:42 -S /tmp/ovirt-id_rsa_c05b1fdf-4784-46e5-8384-fb5ba869c49f -p 80 -b -B rhevm http://guido.lab.eng.blr.redhat.com:80/Components/vds/ http://guido.lab.eng.blr.redhat.com:80/Components/vds/ 10.70.36.15 c05b1fdf-4784-46e5-8384-fb5ba869c49f False' > at org.ovirt.engine.core.utils.ssh.SSHClient.executeCommand(SSHClient.java:442) [engine-utils.jar:] > at org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH.executeCommand(VdsInstallerSSH.java:387) [engine-utils.jar:] > at org.ovirt.engine.core.bll.VdsInstaller.runBootstrapCommand(VdsInstaller.java:217) [engine-bll.jar:] > at org.ovirt.engine.core.bll.VdsInstaller.RunStage(VdsInstaller.java:341) [engine-bll.jar:] > at org.ovirt.engine.core.bll.VdsInstaller.Install(VdsInstaller.java:280) [engine-bll.jar:] > at org.ovirt.engine.core.bll.InstallVdsCommand.executeCommand(InstallVdsCommand.java:110) [engine-bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:825) [engine-bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:916) [engine-bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1300) [engine-bll.jar:] > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:168) [engine-utils.jar:] > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:107) [engine-utils.jar:] > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:931) [engine-bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:285) [engine-bll.jar:] > at org.ovirt.engine.core.bll.MultipleActionsRunner.executeValidatedCommands(MultipleActionsRunner.java:182) [engine-bll.jar:] > at org.ovirt.engine.core.bll.MultipleActionsRunner.RunCommands(MultipleActionsRunner.java:162) [engine-bll.jar:] > at org.ovirt.engine.core.bll.MultipleActionsRunner$1.run(MultipleActionsRunner.java:84) [engine-bll.jar:] > at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalWrapperRunnable.run(ThreadPoolUtil.java:64) [engine-utils.jar:] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_09-icedtea] > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) [rt.jar:1.7.0_09-icedtea] > at java.util.concurrent.FutureTask.run(FutureTask.java:166) [rt.jar:1.7.0_09-icedtea] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [rt.jar:1.7.0_09-icedtea] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [rt.jar:1.7.0_09-icedtea] > at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_09-icedtea] > >2012-11-08 05:52:08,495 ERROR [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-50) [22279fd7] Installation of 10.70.36.15 has failed. Failure details: SSH command failed while executing at host '10.70.36.15', refer to logs for further information. (Stage: Running first installation script on Host) >2012-11-08 05:52:08,495 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-50) [22279fd7] VDS message: SSH command failed while executing at host '10.70.36.15', refer to logs for further information >2012-11-08 05:52:08,495 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-50) [22279fd7] Script ended, result is {1} >2012-11-08 05:52:08,496 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-4-thread-50) [22279fd7] After Installation pool-4-thread-50 >2012-11-08 05:52:08,498 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-50) [22279fd7] START, SetVdsStatusVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, status=InstallFailed, nonOperationalReason=NONE), log id: 70a2fe7c >2012-11-08 05:52:08,503 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-50) [22279fd7] FINISH, SetVdsStatusVDSCommand, log id: 70a2fe7c >2012-11-08 05:54:44,409 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp-/127.0.0.1:8702-13) [2bd42b67] Running command: UpdateVdsCommand internal: false. Entities affected : ID: 96739be4-2990-11e2-8577-52540035ed48 Type: VDS >2012-11-08 05:54:44,437 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp-/127.0.0.1:8702-13) [2bd42b67] START, SetVdsStatusVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, status=Installing, nonOperationalReason=NONE), log id: 7f482c36 >2012-11-08 05:54:44,439 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp-/127.0.0.1:8702-13) [2bd42b67] FINISH, SetVdsStatusVDSCommand, log id: 7f482c36 >2012-11-08 05:54:44,442 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-4-thread-50) [3f64b07] Running command: InstallVdsCommand internal: true. Entities affected : ID: 96739be4-2990-11e2-8577-52540035ed48 Type: VDS >2012-11-08 05:54:44,443 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-4-thread-50) [3f64b07] Before Installation pool-4-thread-50 >2012-11-08 05:54:44,443 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-50) [3f64b07] Installation of 10.70.36.15. Executing installation stage. (Stage: Starting Host installation) >2012-11-08 05:54:44,443 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-50) [3f64b07] Installation of 10.70.36.15. Executing installation stage. (Stage: Connecting to Host) >2012-11-08 05:54:44,506 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-50) [3f64b07] Installation of 10.70.36.15. Received message: <BSTRAP component='RHEV_INSTALL' status='OK' message='Connected to Host 10.70.36.15 with SSH key fingerprint: cc:0f:a3:d7:15:b2:45:74:3a:6b:c6:2b:4b:f5:0d:0d'/>. FYI. (Stage: Connecting to Host) >2012-11-08 05:54:44,540 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-50) [3f64b07] Installation of 10.70.36.15. Successfully connected to server ssh. (Stage: Connecting to Host) >2012-11-08 05:54:44,540 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-50) [3f64b07] Installation of 10.70.36.15. Executing installation stage. (Stage: Get the unique vds id) >2012-11-08 05:54:44,541 INFO [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-4-thread-50) SSH execute 10.70.36.15 'IDFILE=/etc/vdsm/vdsm.id; if [ -r "${IDFILE}" ]; then cat "${IDFILE}"; else UUID="$(dmidecode -s system-uuid 2> /dev/null | sed -e 's/.*Not.*//' )"; if [ -z "${UUID}" ]; then UUID="$(uuidgen 2> /dev/null)" && mkdir -p "$(dirname "${IDFILE}")" && echo "${UUID}" > "${IDFILE}" && chmod 0644 "${IDFILE}"; fi; [ -n "${UUID}" ] && echo "${UUID}"; fi' >2012-11-08 05:54:44,676 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-29) Installation of 10.70.36.15. Received message: 00000000-0000-0000-0000-0025902F7254. FYI. (Stage: Get the unique vds id) >2012-11-08 05:54:44,683 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-29) Installation of 10.70.36.15. Assigning unique id 00000000-0000-0000-0000-0025902F7254 to Host. (Stage: Get the unique vds id) >2012-11-08 05:54:44,686 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-50) [3f64b07] Installation of 10.70.36.15. Executing installation stage. (Stage: Upload Installation script to Host) >2012-11-08 05:54:44,686 INFO [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-4-thread-50) SSH execute 10.70.36.15 'cat > '/tmp/ovirt-id_rsa_37db3bdc-1417-4f89-ad61-bb111d2ebec1'' >2012-11-08 05:54:44,771 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-50) [3f64b07] Installation of 10.70.36.15. Executing installation stage. (Stage: Running first installation script on Host) >2012-11-08 05:54:44,771 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-50) [3f64b07] Installation of 10.70.36.15. Sending SSH Command umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar -C "${MYTMP}" --no-same-permissions -o -x && "${MYTMP}"/setup -c 'ssl=true;management_port=54321' -O 'lab.eng.blr.redhat.com' -t 2012-11-08T10:54:44 -S /tmp/ovirt-id_rsa_37db3bdc-1417-4f89-ad61-bb111d2ebec1 -p 80 -b -B rhevm http://guido.lab.eng.blr.redhat.com:80/Components/vds/ http://guido.lab.eng.blr.redhat.com:80/Components/vds/ 10.70.36.15 37db3bdc-1417-4f89-ad61-bb111d2ebec1 False < /var/cache/ovirt-engine/vdsm-bootstrap-2.tar. (Stage: Running first installation script on Host) >2012-11-08 05:54:44,772 INFO [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-4-thread-50) SSH execute 10.70.36.15 'umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar -C "${MYTMP}" --no-same-permissions -o -x && "${MYTMP}"/setup -c 'ssl=true;management_port=54321' -O 'lab.eng.blr.redhat.com' -t 2012-11-08T10:54:44 -S /tmp/ovirt-id_rsa_37db3bdc-1417-4f89-ad61-bb111d2ebec1 -p 80 -b -B rhevm http://guido.lab.eng.blr.redhat.com:80/Components/vds/ http://guido.lab.eng.blr.redhat.com:80/Components/vds/ 10.70.36.15 37db3bdc-1417-4f89-ad61-bb111d2ebec1 False' >2012-11-08 05:54:44,864 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-29) Installation of 10.70.36.15. Received message: <BSTRAP component='INSTALLER' status='OK' message='Test platform succeeded'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:54:45,950 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-29) Installation of 10.70.36.15. Received message: <BSTRAP component='SET_SYSTEM_TIME' status='OK' message='setSystemTime ended successfully'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:54:48,053 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-29) Installation of 10.70.36.15. Received message: <BSTRAP component='RHN_REGISTRATION' status='OK' message='Host properly registered with RHN/Satellite.'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:54:48,200 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-29) Installation of 10.70.36.15. Received message: <BSTRAP component='VDSM_MAJOR_VER' status='OK' message='Available VDSM matches requirements'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:54:48,204 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-29) Installation of 10.70.36.15. Received message: <BSTRAP component='VT_SVM' status='OK' processor='Intel' message='Server supports virtualization'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:54:48,212 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-29) Installation of 10.70.36.15. Received message: <BSTRAP component='OS' status='OK' type='RHEL6' message='Supported platform version'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:54:48,216 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-29) Installation of 10.70.36.15. Received message: <BSTRAP component='KERNEL' status='OK' version='279' message='Supported kernel version: 279'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:54:48,270 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-29) Installation of 10.70.36.15. Received message: <BSTRAP component='REQ PACKAGES' status='OK' result='SDL.x86_64' message='SDL-1.2.14-3.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:54:48,284 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-29) Installation of 10.70.36.15. Received message: <BSTRAP component='REQ PACKAGES' status='OK' result='bridge-utils.x86_64' message='bridge-utils-1.2-9.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:54:48,298 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-29) Installation of 10.70.36.15. Received message: <BSTRAP component='REQ PACKAGES' status='OK' result='mesa-libGLU.x86_64' message='mesa-libGLU-7.11-5.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:54:48,313 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-29) Installation of 10.70.36.15. Received message: <BSTRAP component='REQ PACKAGES' status='OK' result='openssl.x86_64' message='openssl-1.0.0-25.el6_3.1.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:54:48,328 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-29) Installation of 10.70.36.15. Received message: <BSTRAP component='REQ PACKAGES' status='OK' result='m2crypto.x86_64' message='m2crypto-0.20.2-9.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:54:48,342 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-29) Installation of 10.70.36.15. Received message: <BSTRAP component='REQ PACKAGES' status='OK' result='rsync.x86_64' message='rsync-3.0.6-9.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:54:48,355 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-29) Installation of 10.70.36.15. Received message: <BSTRAP component='VDS PACKAGES' status='WARN' result='qemu-kvm' message='package qemu-kvm is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:54:48,368 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-29) Installation of 10.70.36.15. Received message: <BSTRAP component='VDS PACKAGES' status='WARN' result='qemu-kvm-tools' message='package qemu-kvm-tools is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:54:48,382 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-29) Installation of 10.70.36.15. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='(~vdsm~, ~vdsm-cli~)' message='vdsm-4.9.6-41.0.el6_3.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:54:48,396 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-29) Installation of 10.70.36.15. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='libjpeg' message='libjpeg-6b-46.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:54:48,413 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-29) Installation of 10.70.36.15. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='spice-server' message='spice-server-0.10.1-10.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:54:48,423 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-29) Installation of 10.70.36.15. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='pixman' message='pixman-0.18.4-1.el6_0.1.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:54:48,437 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-29) Installation of 10.70.36.15. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='seabios' message='seabios-0.6.1.2-19.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:54:48,450 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-29) Installation of 10.70.36.15. Received message: <BSTRAP component='VDS PACKAGES' status='WARN' result='qemu-img' message='package qemu-img is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:54:48,464 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-29) Installation of 10.70.36.15. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='fence-agents' message='fence-agents-3.1.5-17.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:54:48,479 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-29) Installation of 10.70.36.15. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='libselinux-python' message='libselinux-python-2.0.94-5.3.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:54:48,492 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-29) Installation of 10.70.36.15. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='sanlock' message='sanlock-2.3-4.el6_3.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:54:48,505 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-29) Installation of 10.70.36.15. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='sanlock-python' message='sanlock-python-2.3-4.el6_3.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:55:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-1) Autorecovering hosts is disabled, skipping >2012-11-08 05:55:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-1) Autorecovering storage domains is disabled, skipping >2012-11-08 05:55:30,310 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-29) Installation of 10.70.36.15. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='(~vdsm~, ~vdsm-cli~)' message='vdsm-4.9.6-41.0.el6_3.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:55:32,537 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-29) Installation of 10.70.36.15. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='libjpeg' message='libjpeg-6b-46.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:55:34,774 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-29) Installation of 10.70.36.15. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='spice-server' message='spice-server-0.10.1-10.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:55:35,045 INFO [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (ajp-/127.0.0.1:8702-2) SSH execute 10.70.36.8 'IDFILE=/etc/vdsm/vdsm.id; if [ -r "${IDFILE}" ]; then cat "${IDFILE}"; else UUID="$(dmidecode -s system-uuid 2> /dev/null | sed -e 's/.*Not.*//' )"; if [ -z "${UUID}" ]; then UUID="$(uuidgen 2> /dev/null)" && mkdir -p "$(dirname "${IDFILE}")" && echo "${UUID}" > "${IDFILE}" && chmod 0644 "${IDFILE}"; fi; [ -n "${UUID}" ] && echo "${UUID}"; fi' >2012-11-08 05:55:35,207 INFO [org.ovirt.engine.core.bll.AddVdsCommand] (ajp-/127.0.0.1:8702-2) [222f859b] Running command: AddVdsCommand internal: false. Entities affected : ID: 6cea47a6-2976-11e2-ad99-52540035ed48 Type: VdsGroups >2012-11-08 05:55:35,223 INFO [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp-/127.0.0.1:8702-2) [1e740a96] Running command: AddVdsSpmIdCommand internal: true. Entities affected : ID: d2bd32d4-2992-11e2-ab59-52540035ed48 Type: VDS >2012-11-08 05:55:35,228 INFO [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp-/127.0.0.1:8702-2) [1e740a96] Lock freed to object EngineLock [exclusiveLocks= key: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 value: REGISTER_VDS >, sharedLocks= ] >2012-11-08 05:55:35,229 ERROR [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp-/127.0.0.1:8702-2) [1e740a96] Cannot get vdsManager for vdsid=d2bd32d4-2992-11e2-ab59-52540035ed48 >2012-11-08 05:55:35,230 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (ajp-/127.0.0.1:8702-2) [1e740a96] START, RemoveVdsVDSCommand(HostName = rhs-gp-srv4.lab.eng.blr.redhat.com, HostId = d2bd32d4-2992-11e2-ab59-52540035ed48), log id: 26555a0f >2012-11-08 05:55:35,230 ERROR [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp-/127.0.0.1:8702-2) [1e740a96] Cannot get vdsManager for vdsid=d2bd32d4-2992-11e2-ab59-52540035ed48 >2012-11-08 05:55:35,230 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (ajp-/127.0.0.1:8702-2) [1e740a96] FINISH, RemoveVdsVDSCommand, log id: 26555a0f >2012-11-08 05:55:35,230 ERROR [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp-/127.0.0.1:8702-2) [1e740a96] Cannot get vdsManager for vdsid=d2bd32d4-2992-11e2-ab59-52540035ed48 >2012-11-08 05:55:35,231 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-2) [1e740a96] START, AddVdsVDSCommand(HostName = rhs-gp-srv4.lab.eng.blr.redhat.com, HostId = d2bd32d4-2992-11e2-ab59-52540035ed48), log id: 4803b017 >2012-11-08 05:55:35,231 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-2) [1e740a96] AddVds - entered , starting logic to add VDS d2bd32d4-2992-11e2-ab59-52540035ed48 >2012-11-08 05:55:35,233 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-2) [1e740a96] AddVds - VDS d2bd32d4-2992-11e2-ab59-52540035ed48 was added, will try to add it to the resource manager >2012-11-08 05:55:35,233 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ajp-/127.0.0.1:8702-2) [1e740a96] Eneterd VdsManager:constructor >2012-11-08 05:55:35,234 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ajp-/127.0.0.1:8702-2) [1e740a96] vdsBroker(10.70.36.8,54,321) >2012-11-08 05:55:35,235 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp-/127.0.0.1:8702-2) [1e740a96] ResourceManager::AddVds - VDS d2bd32d4-2992-11e2-ab59-52540035ed48 was added to the Resource Manager >2012-11-08 05:55:35,235 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-2) [1e740a96] FINISH, AddVdsVDSCommand, log id: 4803b017 >2012-11-08 05:55:35,250 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-4-thread-49) [28d83cc1] Running command: InstallVdsCommand internal: true. Entities affected : ID: d2bd32d4-2992-11e2-ab59-52540035ed48 Type: VDS >2012-11-08 05:55:35,251 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-4-thread-49) [28d83cc1] Before Installation pool-4-thread-49 >2012-11-08 05:55:35,251 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-49) [28d83cc1] Installation of 10.70.36.8. Executing installation stage. (Stage: Starting Host installation) >2012-11-08 05:55:35,251 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-49) [28d83cc1] Installation of 10.70.36.8. Executing installation stage. (Stage: Connecting to Host) >2012-11-08 05:55:35,313 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-49) [28d83cc1] Installation of 10.70.36.8. Received message: <BSTRAP component='RHEV_INSTALL' status='OK' message='Connected to Host 10.70.36.8 with SSH key fingerprint: be:b4:2c:77:3c:6e:60:f7:35:69:fb:29:52:29:15:e1'/>. FYI. (Stage: Connecting to Host) >2012-11-08 05:55:35,350 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-49) [28d83cc1] Installation of 10.70.36.8. Successfully connected to server ssh. (Stage: Connecting to Host) >2012-11-08 05:55:35,350 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-49) [28d83cc1] Installation of 10.70.36.8. Executing installation stage. (Stage: Get the unique vds id) >2012-11-08 05:55:35,350 INFO [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-4-thread-49) SSH execute 10.70.36.8 'IDFILE=/etc/vdsm/vdsm.id; if [ -r "${IDFILE}" ]; then cat "${IDFILE}"; else UUID="$(dmidecode -s system-uuid 2> /dev/null | sed -e 's/.*Not.*//' )"; if [ -z "${UUID}" ]; then UUID="$(uuidgen 2> /dev/null)" && mkdir -p "$(dirname "${IDFILE}")" && echo "${UUID}" > "${IDFILE}" && chmod 0644 "${IDFILE}"; fi; [ -n "${UUID}" ] && echo "${UUID}"; fi' >2012-11-08 05:55:35,474 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-34) Installation of 10.70.36.8. Received message: 00000000-0000-0000-0000-0025902F8CA8. FYI. (Stage: Get the unique vds id) >2012-11-08 05:55:35,476 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-34) Installation of 10.70.36.8. Assigning unique id 00000000-0000-0000-0000-0025902F8CA8 to Host. (Stage: Get the unique vds id) >2012-11-08 05:55:35,479 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-49) [28d83cc1] Installation of 10.70.36.8. Executing installation stage. (Stage: Upload Installation script to Host) >2012-11-08 05:55:35,479 INFO [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-4-thread-49) SSH execute 10.70.36.8 'cat > '/tmp/ovirt-id_rsa_c235127a-605a-4495-b0ed-8184ab5c03b5'' >2012-11-08 05:55:35,564 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-49) [28d83cc1] Installation of 10.70.36.8. Executing installation stage. (Stage: Running first installation script on Host) >2012-11-08 05:55:35,564 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-49) [28d83cc1] Installation of 10.70.36.8. Sending SSH Command umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar -C "${MYTMP}" --no-same-permissions -o -x && "${MYTMP}"/setup -c 'ssl=true;management_port=54321' -O 'lab.eng.blr.redhat.com' -t 2012-11-08T10:55:35 -S /tmp/ovirt-id_rsa_c235127a-605a-4495-b0ed-8184ab5c03b5 -p 80 -b -B rhevm http://guido.lab.eng.blr.redhat.com:80/Components/vds/ http://guido.lab.eng.blr.redhat.com:80/Components/vds/ 10.70.36.8 c235127a-605a-4495-b0ed-8184ab5c03b5 False < /var/cache/ovirt-engine/vdsm-bootstrap-2.tar. (Stage: Running first installation script on Host) >2012-11-08 05:55:35,565 INFO [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-4-thread-49) SSH execute 10.70.36.8 'umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar -C "${MYTMP}" --no-same-permissions -o -x && "${MYTMP}"/setup -c 'ssl=true;management_port=54321' -O 'lab.eng.blr.redhat.com' -t 2012-11-08T10:55:35 -S /tmp/ovirt-id_rsa_c235127a-605a-4495-b0ed-8184ab5c03b5 -p 80 -b -B rhevm http://guido.lab.eng.blr.redhat.com:80/Components/vds/ http://guido.lab.eng.blr.redhat.com:80/Components/vds/ 10.70.36.8 c235127a-605a-4495-b0ed-8184ab5c03b5 False' >2012-11-08 05:55:35,672 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-34) Installation of 10.70.36.8. Received message: <BSTRAP component='INSTALLER' status='OK' message='Test platform succeeded'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:55:36,773 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-34) Installation of 10.70.36.8. Received message: <BSTRAP component='SET_SYSTEM_TIME' status='OK' message='setSystemTime ended successfully'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:55:37,012 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-29) Installation of 10.70.36.15. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='pixman' message='pixman-0.18.4-1.el6_0.1.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:55:37,237 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-47) Initializing Host: rhs-gp-srv4.lab.eng.blr.redhat.com >2012-11-08 05:55:38,568 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-34) Installation of 10.70.36.8. Received message: <BSTRAP component='RHN_REGISTRATION' status='OK' message='Host properly registered with RHN/Satellite.'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:55:38,737 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-34) Installation of 10.70.36.8. Received message: <BSTRAP component='VDSM_MAJOR_VER' status='OK' message='Available VDSM matches requirements'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:55:38,744 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-34) Installation of 10.70.36.8. Received message: <BSTRAP component='VT_SVM' status='OK' processor='Intel' message='Server supports virtualization'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:55:38,748 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-34) Installation of 10.70.36.8. Received message: <BSTRAP component='OS' status='OK' type='RHEL6' message='Supported platform version'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:55:38,753 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-34) Installation of 10.70.36.8. Received message: <BSTRAP component='KERNEL' status='OK' version='279' message='Supported kernel version: 279'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:55:38,804 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-34) Installation of 10.70.36.8. Received message: <BSTRAP component='REQ PACKAGES' status='WARN' result='SDL.x86_64' message='package SDL.x86_64 is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:55:38,823 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-34) Installation of 10.70.36.8. Received message: <BSTRAP component='REQ PACKAGES' status='OK' result='bridge-utils.x86_64' message='bridge-utils-1.2-9.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:55:38,834 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-34) Installation of 10.70.36.8. Received message: <BSTRAP component='REQ PACKAGES' status='WARN' result='mesa-libGLU.x86_64' message='package mesa-libGLU.x86_64 is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:55:38,849 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-34) Installation of 10.70.36.8. Received message: <BSTRAP component='REQ PACKAGES' status='OK' result='openssl.x86_64' message='openssl-1.0.0-25.el6_3.1.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:55:38,864 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-34) Installation of 10.70.36.8. Received message: <BSTRAP component='REQ PACKAGES' status='OK' result='m2crypto.x86_64' message='m2crypto-0.20.2-9.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:55:38,877 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-34) Installation of 10.70.36.8. Received message: <BSTRAP component='REQ PACKAGES' status='WARN' result='rsync.x86_64' message='package rsync.x86_64 is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:55:38,890 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-34) Installation of 10.70.36.8. Received message: <BSTRAP component='VDS PACKAGES' status='WARN' result='qemu-kvm' message='package qemu-kvm is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:55:38,903 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-34) Installation of 10.70.36.8. Received message: <BSTRAP component='VDS PACKAGES' status='WARN' result='qemu-kvm-tools' message='package qemu-kvm-tools is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:55:38,917 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-34) Installation of 10.70.36.8. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='(~vdsm~, ~vdsm-cli~)' message='vdsm-4.9.6-41.0.el6_3.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:55:38,931 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-34) Installation of 10.70.36.8. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='libjpeg' message='libjpeg-6b-46.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:55:38,945 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-34) Installation of 10.70.36.8. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='spice-server' message='spice-server-0.10.1-10.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:55:38,959 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-34) Installation of 10.70.36.8. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='pixman' message='pixman-0.18.4-1.el6_0.1.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:55:38,973 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-34) Installation of 10.70.36.8. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='seabios' message='seabios-0.6.1.2-19.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:55:38,986 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-34) Installation of 10.70.36.8. Received message: <BSTRAP component='VDS PACKAGES' status='WARN' result='qemu-img' message='package qemu-img is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:55:39,000 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-34) Installation of 10.70.36.8. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='fence-agents' message='fence-agents-3.1.5-17.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:55:39,014 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-34) Installation of 10.70.36.8. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='libselinux-python' message='libselinux-python-2.0.94-5.3.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:55:39,028 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-34) Installation of 10.70.36.8. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='sanlock' message='sanlock-2.3-4.el6_3.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:55:39,041 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-34) Installation of 10.70.36.8. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='sanlock-python' message='sanlock-python-2.3-4.el6_3.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:55:39,212 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-29) Installation of 10.70.36.15. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='seabios' message='seabios-0.6.1.2-19.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:55:41,488 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-29) Installation of 10.70.36.15. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='fence-agents' message='fence-agents-3.1.5-17.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:55:43,835 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-29) Installation of 10.70.36.15. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='libselinux-python' message='libselinux-python-2.0.94-5.3.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:55:46,049 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-29) Installation of 10.70.36.15. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='sanlock' message='sanlock-2.3-4.el6_3.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:55:48,608 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-29) Installation of 10.70.36.15. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='sanlock-python' message='sanlock-python-2.3-4.el6_3.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:55:50,868 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-29) Installation of 10.70.36.15. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='qemu-img' message='package qemu-img is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:55:53,122 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-29) Installation of 10.70.36.15. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='qemu-kvm-tools' message='package qemu-kvm-tools is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:55:55,359 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-29) Installation of 10.70.36.15. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='qemu-kvm' message='package qemu-kvm is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:55:55,369 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-29) Installation of 10.70.36.15. Received message: <BSTRAP component='CreateConf' status='OK' message='Basic configuration found, skipping this step'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:55:55,384 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-29) Installation of 10.70.36.15. Received message: <BSTRAP component='CheckLocalHostname' status='OK' message='Local hostname is correct.'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:56:05,783 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-29) Installation of 10.70.36.15. Received message: <BSTRAP component='SetNetworking' status='FAIL' message='addNetwork error trying to add management bridge'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:56:05,788 ERROR [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-29) Installation of 10.70.36.15. Received message: <BSTRAP component='RHEV_INSTALL' status='FAIL'/>. Error occured. (Stage: Running first installation script on Host) >2012-11-08 05:56:05,817 ERROR [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-50) [3f64b07] Installation of 10.70.36.15. Error: Repository JBOSS is listed more than once in the configuration >Repository qa-latest-5 is listed more than once in the configuration >Repository rhel-source is listed more than once in the configuration >Repository rhel-source-beta is listed more than once in the configuration >Repository rhs-client is listed more than once in the configuration >Repository JBOSS is listed more than once in the configuration >Repository qa-latest-5 is listed more than once in the configuration >Repository rhel-source is listed more than once in the configuration >Repository rhel-source-beta is listed more than once in the configuration >Repository rhs-client is listed more than once in the configuration >. (Stage: Running first installation script on Host) >2012-11-08 05:56:05,817 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-50) [3f64b07] VDS message: Repository JBOSS is listed more than once in the configuration >2012-11-08 05:56:05,817 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-50) [3f64b07] VDS message: Repository qa-latest-5 is listed more than once in the configuration >2012-11-08 05:56:05,817 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-50) [3f64b07] VDS message: Repository rhel-source is listed more than once in the configuration >2012-11-08 05:56:05,817 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-50) [3f64b07] VDS message: Repository rhel-source-beta is listed more than once in the configuration >2012-11-08 05:56:05,817 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-50) [3f64b07] VDS message: Repository rhs-client is listed more than once in the configuration >2012-11-08 05:56:05,817 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-50) [3f64b07] VDS message: Repository JBOSS is listed more than once in the configuration >2012-11-08 05:56:05,817 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-50) [3f64b07] VDS message: Repository qa-latest-5 is listed more than once in the configuration >2012-11-08 05:56:05,817 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-50) [3f64b07] VDS message: Repository rhel-source is listed more than once in the configuration >2012-11-08 05:56:05,817 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-50) [3f64b07] VDS message: Repository rhel-source-beta is listed more than once in the configuration >2012-11-08 05:56:05,817 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-50) [3f64b07] VDS message: Repository rhs-client is listed more than once in the configuration >2012-11-08 05:56:05,817 ERROR [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-4-thread-50) SSH error running command 10.70.36.15:'umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar -C "${MYTMP}" --no-same-permissions -o -x && "${MYTMP}"/setup -c 'ssl=true;management_port=54321' -O 'lab.eng.blr.redhat.com' -t 2012-11-08T10:54:44 -S /tmp/ovirt-id_rsa_37db3bdc-1417-4f89-ad61-bb111d2ebec1 -p 80 -b -B rhevm http://guido.lab.eng.blr.redhat.com:80/Components/vds/ http://guido.lab.eng.blr.redhat.com:80/Components/vds/ 10.70.36.15 37db3bdc-1417-4f89-ad61-bb111d2ebec1 False': java.io.IOException: Command returned failure code 1 during SSH session 10.70.36.15:22' 'umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar -C "${MYTMP}" --no-same-permissions -o -x && "${MYTMP}"/setup -c 'ssl=true;management_port=54321' -O 'lab.eng.blr.redhat.com' -t 2012-11-08T10:54:44 -S /tmp/ovirt-id_rsa_37db3bdc-1417-4f89-ad61-bb111d2ebec1 -p 80 -b -B rhevm http://guido.lab.eng.blr.redhat.com:80/Components/vds/ http://guido.lab.eng.blr.redhat.com:80/Components/vds/ 10.70.36.15 37db3bdc-1417-4f89-ad61-bb111d2ebec1 False' > at org.ovirt.engine.core.utils.ssh.SSHClient.executeCommand(SSHClient.java:442) [engine-utils.jar:] > at org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH.executeCommand(VdsInstallerSSH.java:387) [engine-utils.jar:] > at org.ovirt.engine.core.bll.VdsInstaller.runBootstrapCommand(VdsInstaller.java:217) [engine-bll.jar:] > at org.ovirt.engine.core.bll.VdsInstaller.RunStage(VdsInstaller.java:341) [engine-bll.jar:] > at org.ovirt.engine.core.bll.VdsInstaller.Install(VdsInstaller.java:280) [engine-bll.jar:] > at org.ovirt.engine.core.bll.InstallVdsCommand.executeCommand(InstallVdsCommand.java:110) [engine-bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:825) [engine-bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:916) [engine-bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1300) [engine-bll.jar:] > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:168) [engine-utils.jar:] > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:107) [engine-utils.jar:] > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:931) [engine-bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:285) [engine-bll.jar:] > at org.ovirt.engine.core.bll.MultipleActionsRunner.executeValidatedCommands(MultipleActionsRunner.java:182) [engine-bll.jar:] > at org.ovirt.engine.core.bll.MultipleActionsRunner.RunCommands(MultipleActionsRunner.java:162) [engine-bll.jar:] > at org.ovirt.engine.core.bll.MultipleActionsRunner$1.run(MultipleActionsRunner.java:84) [engine-bll.jar:] > at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalWrapperRunnable.run(ThreadPoolUtil.java:64) [engine-utils.jar:] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_09-icedtea] > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) [rt.jar:1.7.0_09-icedtea] > at java.util.concurrent.FutureTask.run(FutureTask.java:166) [rt.jar:1.7.0_09-icedtea] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [rt.jar:1.7.0_09-icedtea] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [rt.jar:1.7.0_09-icedtea] > at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_09-icedtea] > >2012-11-08 05:56:05,818 ERROR [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-50) [3f64b07] Installation of 10.70.36.15 has failed. Failure details: SSH command failed while executing at host '10.70.36.15', refer to logs for further information. (Stage: Running first installation script on Host) >2012-11-08 05:56:05,818 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-50) [3f64b07] VDS message: SSH command failed while executing at host '10.70.36.15', refer to logs for further information >2012-11-08 05:56:05,818 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-50) [3f64b07] Script ended, result is {1} >2012-11-08 05:56:05,821 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-4-thread-50) [3f64b07] After Installation pool-4-thread-50 >2012-11-08 05:56:05,823 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-50) [3f64b07] START, SetVdsStatusVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, status=InstallFailed, nonOperationalReason=NONE), log id: 48bc8923 >2012-11-08 05:56:05,827 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-50) [3f64b07] FINISH, SetVdsStatusVDSCommand, log id: 48bc8923 >2012-11-08 05:56:42,275 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-34) Installation of 10.70.36.8. Received message: <BSTRAP component='REQ PACKAGES' status='OK' result='rsync.x86_64' message='rsync-3.0.6-9.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:57:30,654 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-34) Installation of 10.70.36.8. Received message: <BSTRAP component='REQ PACKAGES' status='OK' result='mesa-libGLU.x86_64' message='mesa-libGLU-7.11-5.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:57:44,563 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-34) Installation of 10.70.36.8. Received message: <BSTRAP component='REQ PACKAGES' status='OK' result='SDL.x86_64' message='SDL-1.2.14-3.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:57:47,388 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-34) Installation of 10.70.36.8. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='(~vdsm~, ~vdsm-cli~)' message='vdsm-4.9.6-41.0.el6_3.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:57:49,743 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-34) Installation of 10.70.36.8. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='libjpeg' message='libjpeg-6b-46.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:57:52,149 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-34) Installation of 10.70.36.8. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='spice-server' message='spice-server-0.10.1-10.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:57:54,657 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-34) Installation of 10.70.36.8. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='pixman' message='pixman-0.18.4-1.el6_0.1.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:57:57,100 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-34) Installation of 10.70.36.8. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='seabios' message='seabios-0.6.1.2-19.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:57:59,917 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-34) Installation of 10.70.36.8. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='fence-agents' message='fence-agents-3.1.5-17.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:58:02,324 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-34) Installation of 10.70.36.8. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='libselinux-python' message='libselinux-python-2.0.94-5.3.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:58:04,698 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-34) Installation of 10.70.36.8. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='sanlock' message='sanlock-2.3-4.el6_3.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:58:07,100 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-34) Installation of 10.70.36.8. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='sanlock-python' message='sanlock-python-2.3-4.el6_3.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:58:09,510 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-34) Installation of 10.70.36.8. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='qemu-img' message='package qemu-img is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:58:23,260 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-34) Installation of 10.70.36.8. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='qemu-kvm-tools' message='package qemu-kvm-tools is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:58:25,678 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-34) Installation of 10.70.36.8. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='qemu-kvm' message='package qemu-kvm is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:58:25,684 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-34) Installation of 10.70.36.8. Received message: <BSTRAP component='CreateConf' status='OK' message='Basic configuration set'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:58:25,693 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-34) Installation of 10.70.36.8. Received message: <BSTRAP component='CheckLocalHostname' status='OK' message='Local hostname is correct.'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:58:36,161 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-34) Installation of 10.70.36.8. Received message: <BSTRAP component='SetNetworking' status='FAIL' message='addNetwork error trying to add management bridge'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 05:58:36,166 ERROR [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-34) Installation of 10.70.36.8. Received message: <BSTRAP component='RHEV_INSTALL' status='FAIL'/>. Error occured. (Stage: Running first installation script on Host) >2012-11-08 05:58:36,196 ERROR [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-49) [28d83cc1] Installation of 10.70.36.8. Error: Repository JBOSS is listed more than once in the configuration >Repository qa-latest-5 is listed more than once in the configuration >Repository rhel-source is listed more than once in the configuration >Repository rhel-source-beta is listed more than once in the configuration >Repository rhs-client is listed more than once in the configuration >Repository JBOSS is listed more than once in the configuration >Repository qa-latest-5 is listed more than once in the configuration >Repository rhel-source is listed more than once in the configuration >Repository rhel-source-beta is listed more than once in the configuration >Repository rhs-client is listed more than once in the configuration >. (Stage: Running first installation script on Host) >2012-11-08 05:58:36,196 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-49) [28d83cc1] VDS message: Repository JBOSS is listed more than once in the configuration >2012-11-08 05:58:36,196 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-49) [28d83cc1] VDS message: Repository qa-latest-5 is listed more than once in the configuration >2012-11-08 05:58:36,196 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-49) [28d83cc1] VDS message: Repository rhel-source is listed more than once in the configuration >2012-11-08 05:58:36,197 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-49) [28d83cc1] VDS message: Repository rhel-source-beta is listed more than once in the configuration >2012-11-08 05:58:36,197 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-49) [28d83cc1] VDS message: Repository rhs-client is listed more than once in the configuration >2012-11-08 05:58:36,197 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-49) [28d83cc1] VDS message: Repository JBOSS is listed more than once in the configuration >2012-11-08 05:58:36,197 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-49) [28d83cc1] VDS message: Repository qa-latest-5 is listed more than once in the configuration >2012-11-08 05:58:36,197 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-49) [28d83cc1] VDS message: Repository rhel-source is listed more than once in the configuration >2012-11-08 05:58:36,197 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-49) [28d83cc1] VDS message: Repository rhel-source-beta is listed more than once in the configuration >2012-11-08 05:58:36,197 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-49) [28d83cc1] VDS message: Repository rhs-client is listed more than once in the configuration >2012-11-08 05:58:36,197 ERROR [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-4-thread-49) SSH error running command 10.70.36.8:'umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar -C "${MYTMP}" --no-same-permissions -o -x && "${MYTMP}"/setup -c 'ssl=true;management_port=54321' -O 'lab.eng.blr.redhat.com' -t 2012-11-08T10:55:35 -S /tmp/ovirt-id_rsa_c235127a-605a-4495-b0ed-8184ab5c03b5 -p 80 -b -B rhevm http://guido.lab.eng.blr.redhat.com:80/Components/vds/ http://guido.lab.eng.blr.redhat.com:80/Components/vds/ 10.70.36.8 c235127a-605a-4495-b0ed-8184ab5c03b5 False': java.io.IOException: Command returned failure code 1 during SSH session 10.70.36.8:22' 'umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar -C "${MYTMP}" --no-same-permissions -o -x && "${MYTMP}"/setup -c 'ssl=true;management_port=54321' -O 'lab.eng.blr.redhat.com' -t 2012-11-08T10:55:35 -S /tmp/ovirt-id_rsa_c235127a-605a-4495-b0ed-8184ab5c03b5 -p 80 -b -B rhevm http://guido.lab.eng.blr.redhat.com:80/Components/vds/ http://guido.lab.eng.blr.redhat.com:80/Components/vds/ 10.70.36.8 c235127a-605a-4495-b0ed-8184ab5c03b5 False' > at org.ovirt.engine.core.utils.ssh.SSHClient.executeCommand(SSHClient.java:442) [engine-utils.jar:] > at org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH.executeCommand(VdsInstallerSSH.java:387) [engine-utils.jar:] > at org.ovirt.engine.core.bll.VdsInstaller.runBootstrapCommand(VdsInstaller.java:217) [engine-bll.jar:] > at org.ovirt.engine.core.bll.VdsInstaller.RunStage(VdsInstaller.java:341) [engine-bll.jar:] > at org.ovirt.engine.core.bll.VdsInstaller.Install(VdsInstaller.java:280) [engine-bll.jar:] > at org.ovirt.engine.core.bll.InstallVdsCommand.executeCommand(InstallVdsCommand.java:110) [engine-bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:825) [engine-bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:916) [engine-bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1300) [engine-bll.jar:] > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:168) [engine-utils.jar:] > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:107) [engine-utils.jar:] > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:931) [engine-bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:285) [engine-bll.jar:] > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:336) [engine-bll.jar:] > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:593) [engine-bll.jar:] > at sun.reflect.GeneratedMethodAccessor136.invoke(Unknown Source) [:1.7.0_09-icedtea] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_09-icedtea] > at java.lang.reflect.Method.invoke(Method.java:601) [rt.jar:1.7.0_09-icedtea] > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) [jboss-as-ee.jar:7.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:36) [jboss-as-ee.jar:7.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) [jboss-as-ejb3.jar:7.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) [jboss-as-ee.jar:7.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:53) [jboss-as-ejb3.jar:7.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:210) [jboss-as-ejb3.jar:7.1.2.Final-redhat-1] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:362) [jboss-as-ejb3.jar:7.1.2.Final-redhat-1] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:193) [jboss-as-ejb3.jar:7.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [jboss-as-ejb3.jar:7.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:42) [jboss-as-ejb3.jar:7.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) [jboss-as-ejb3.jar:7.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) [jboss-as-ee.jar:7.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) [jboss-as-ee.jar:7.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) [jboss-as-ee.jar:7.1.2.Final-redhat-1] > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:176) [jboss-as-ee.jar:7.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) [jboss-as-ee.jar:7.1.2.Final-redhat-1] > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) [engine-bll.jar:] > at org.ovirt.engine.core.bll.AddVdsCommand$3.run(AddVdsCommand.java:156) [engine-bll.jar:] > at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalWrapperRunnable.run(ThreadPoolUtil.java:64) [engine-utils.jar:] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_09-icedtea] > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) [rt.jar:1.7.0_09-icedtea] > at java.util.concurrent.FutureTask.run(FutureTask.java:166) [rt.jar:1.7.0_09-icedtea] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [rt.jar:1.7.0_09-icedtea] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [rt.jar:1.7.0_09-icedtea] > at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_09-icedtea] > >2012-11-08 05:58:36,199 ERROR [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-49) [28d83cc1] Installation of 10.70.36.8 has failed. Failure details: SSH command failed while executing at host '10.70.36.8', refer to logs for further information. (Stage: Running first installation script on Host) >2012-11-08 05:58:36,199 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-49) [28d83cc1] VDS message: SSH command failed while executing at host '10.70.36.8', refer to logs for further information >2012-11-08 05:58:36,199 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-49) [28d83cc1] Script ended, result is {1} >2012-11-08 05:58:36,200 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-4-thread-49) [28d83cc1] After Installation pool-4-thread-49 >2012-11-08 05:58:36,202 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-49) [28d83cc1] START, SetVdsStatusVDSCommand(HostName = rhs-gp-srv4.lab.eng.blr.redhat.com, HostId = d2bd32d4-2992-11e2-ab59-52540035ed48, status=InstallFailed, nonOperationalReason=NONE), log id: c6faa0c >2012-11-08 05:58:36,206 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-49) [28d83cc1] FINISH, SetVdsStatusVDSCommand, log id: c6faa0c >2012-11-08 05:59:51,867 INFO [org.ovirt.engine.core.bll.MaintananceNumberOfVdssCommand] (pool-4-thread-49) [12337720] Running command: MaintananceNumberOfVdssCommand internal: false. Entities affected : ID: 96739be4-2990-11e2-8577-52540035ed48 Type: VDS >2012-11-08 05:59:51,869 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-49) [12337720] START, SetVdsStatusVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, status=PreparingForMaintenance, nonOperationalReason=NONE), log id: 35b919be >2012-11-08 05:59:51,888 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-49) [12337720] FINISH, SetVdsStatusVDSCommand, log id: 35b919be >2012-11-08 05:59:51,899 INFO [org.ovirt.engine.core.bll.MaintananceVdsCommand] (pool-4-thread-49) [12337720] Running command: MaintananceVdsCommand internal: true. Entities affected : ID: 96739be4-2990-11e2-8577-52540035ed48 Type: VDS >2012-11-08 05:59:53,676 ERROR [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-95) VDS::handleNetworkException Server failed to respond, vds_id = 96739be4-2990-11e2-8577-52540035ed48, vds_name = rhs-gp-srv11.lab.eng.blr.redhat.com, error = VDSNetworkException: >2012-11-08 05:59:53,690 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-95) vds::refreshVdsStats Failed getVdsStats, vds = 96739be4-2990-11e2-8577-52540035ed48 : rhs-gp-srv11.lab.eng.blr.redhat.com, error = VDSNetworkException: VDSNetworkException: >2012-11-08 05:59:53,690 INFO [org.ovirt.engine.core.bll.VdsEventListener] (pool-4-thread-49) ResourceManager::vdsNotResponding entered for Host 96739be4-2990-11e2-8577-52540035ed48, 10.70.36.15 >2012-11-08 05:59:53,697 WARN [org.ovirt.engine.core.bll.VdsNotRespondingTreatmentCommand] (pool-4-thread-49) [501f88bc] CanDoAction of action VdsNotRespondingTreatment failed. Reasons:VDS_FENCING_DISABLED >2012-11-08 06:00:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-9) Autorecovering hosts is disabled, skipping >2012-11-08 06:00:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-9) Autorecovering storage domains is disabled, skipping >2012-11-08 06:00:04,815 INFO [org.ovirt.engine.core.bll.MaintananceNumberOfVdssCommand] (pool-4-thread-49) [1165f9db] Running command: MaintananceNumberOfVdssCommand internal: false. Entities affected : ID: 96739be4-2990-11e2-8577-52540035ed48 Type: VDS >2012-11-08 06:00:04,826 INFO [org.ovirt.engine.core.bll.MaintananceVdsCommand] (pool-4-thread-49) [1165f9db] Running command: MaintananceVdsCommand internal: true. Entities affected : ID: 96739be4-2990-11e2-8577-52540035ed48 Type: VDS >2012-11-08 06:00:04,828 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-49) [1165f9db] START, SetVdsStatusVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, status=Maintenance, nonOperationalReason=NONE), log id: 5811eae3 >2012-11-08 06:00:04,830 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-49) [1165f9db] FINISH, SetVdsStatusVDSCommand, log id: 5811eae3 >2012-11-08 06:00:56,696 INFO [org.ovirt.engine.core.bll.MaintananceNumberOfVdssCommand] (pool-4-thread-49) [529bb1ec] Running command: MaintananceNumberOfVdssCommand internal: false. Entities affected : ID: d2bd32d4-2992-11e2-ab59-52540035ed48 Type: VDS >2012-11-08 06:00:56,697 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-49) [529bb1ec] START, SetVdsStatusVDSCommand(HostName = rhs-gp-srv4.lab.eng.blr.redhat.com, HostId = d2bd32d4-2992-11e2-ab59-52540035ed48, status=PreparingForMaintenance, nonOperationalReason=NONE), log id: 17403c9b >2012-11-08 06:00:56,702 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-49) [529bb1ec] FINISH, SetVdsStatusVDSCommand, log id: 17403c9b >2012-11-08 06:00:56,717 INFO [org.ovirt.engine.core.bll.MaintananceVdsCommand] (pool-4-thread-49) [529bb1ec] Running command: MaintananceVdsCommand internal: true. Entities affected : ID: d2bd32d4-2992-11e2-ab59-52540035ed48 Type: VDS >2012-11-08 06:02:31,751 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-9) XML RPC error in command GetStatsVDS ( HostName = rhs-gp-srv4.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target >2012-11-08 06:02:31,751 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-9) vds::refreshVdsStats Failed getVdsStats, vds = d2bd32d4-2992-11e2-ab59-52540035ed48 : rhs-gp-srv4.lab.eng.blr.redhat.com, error = VDSNetworkException: VDSNetworkException: >2012-11-08 06:02:31,751 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-9) ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = d2bd32d4-2992-11e2-ab59-52540035ed48 : rhs-gp-srv4.lab.eng.blr.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-11-08 06:02:33,767 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-32) XML RPC error in command GetStatsVDS ( HostName = rhs-gp-srv4.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target >2012-11-08 06:02:33,767 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-32) vds::refreshVdsStats Failed getVdsStats, vds = d2bd32d4-2992-11e2-ab59-52540035ed48 : rhs-gp-srv4.lab.eng.blr.redhat.com, error = VDSNetworkException: VDSNetworkException: >2012-11-08 06:02:33,767 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-32) ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = d2bd32d4-2992-11e2-ab59-52540035ed48 : rhs-gp-srv4.lab.eng.blr.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-11-08 06:02:35,779 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-34) XML RPC error in command GetStatsVDS ( HostName = rhs-gp-srv4.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target >2012-11-08 06:02:35,779 ERROR [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-34) VDS::handleNetworkException Server failed to respond, vds_id = d2bd32d4-2992-11e2-ab59-52540035ed48, vds_name = rhs-gp-srv4.lab.eng.blr.redhat.com, error = VDSNetworkException: >2012-11-08 06:02:35,791 INFO [org.ovirt.engine.core.bll.VdsEventListener] (pool-4-thread-49) ResourceManager::vdsNotResponding entered for Host d2bd32d4-2992-11e2-ab59-52540035ed48, 10.70.36.8 >2012-11-08 06:02:35,791 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-34) vds::refreshVdsStats Failed getVdsStats, vds = d2bd32d4-2992-11e2-ab59-52540035ed48 : rhs-gp-srv4.lab.eng.blr.redhat.com, error = VDSNetworkException: VDSNetworkException: >2012-11-08 06:02:35,802 WARN [org.ovirt.engine.core.bll.VdsNotRespondingTreatmentCommand] (pool-4-thread-49) [2f9dd03a] CanDoAction of action VdsNotRespondingTreatment failed. Reasons:VDS_FENCING_DISABLED >2012-11-08 06:02:37,807 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-39) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv4.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target >2012-11-08 06:02:39,826 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-42) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv4.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target >2012-11-08 06:02:41,845 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-44) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv4.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target >2012-11-08 06:02:43,860 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-48) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv4.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target >2012-11-08 06:02:45,890 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-51) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv4.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target >2012-11-08 06:02:47,916 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-55) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv4.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target >2012-11-08 06:02:49,931 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-59) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv4.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target >2012-11-08 06:02:51,955 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-62) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv4.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target >2012-11-08 06:02:53,971 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-67) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv4.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target >2012-11-08 06:02:55,985 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-69) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv4.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target >2012-11-08 06:02:57,999 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-75) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv4.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target >2012-11-08 06:03:00,014 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-79) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv4.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target >2012-11-08 06:03:02,031 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-82) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv4.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target >2012-11-08 06:03:04,051 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-85) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv4.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target >2012-11-08 06:03:06,069 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-88) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv4.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target >2012-11-08 06:03:08,083 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-92) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv4.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target >2012-11-08 06:03:10,103 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-96) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv4.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target >2012-11-08 06:03:12,118 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-99) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv4.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target >2012-11-08 06:03:14,131 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-2) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv4.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target >2012-11-08 06:03:16,147 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-3) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv4.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target >2012-11-08 06:03:18,163 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-6) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv4.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target >2012-11-08 06:03:20,179 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-14) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv4.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target >2012-11-08 06:03:22,194 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-15) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv4.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target >2012-11-08 06:03:24,210 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-20) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv4.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target >2012-11-08 06:03:26,227 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-24) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv4.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target >2012-11-08 06:03:28,245 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-29) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv4.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target >2012-11-08 06:03:30,260 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-32) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv4.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target >2012-11-08 06:03:32,281 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-34) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv4.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target >2012-11-08 06:03:34,296 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-37) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv4.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target >2012-11-08 06:03:36,312 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-38) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv4.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target >2012-11-08 06:03:38,327 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-45) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv4.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target >2012-11-08 06:03:40,348 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-48) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv4.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target >2012-11-08 06:03:42,366 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-51) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv4.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target >2012-11-08 06:03:44,378 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-54) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv4.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target >2012-11-08 06:03:46,400 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-57) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv4.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target >2012-11-08 06:03:48,418 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-61) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv4.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target >2012-11-08 06:03:50,439 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-65) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv4.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target >2012-11-08 06:03:52,456 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-68) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv4.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target >2012-11-08 06:03:54,470 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-71) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv4.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target >2012-11-08 06:03:56,496 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-75) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv4.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target >2012-11-08 06:03:58,507 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-80) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv4.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target >2012-11-08 06:04:00,519 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-84) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv4.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target >2012-11-08 06:04:02,537 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-87) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv4.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target >2012-11-08 06:04:04,552 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-90) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv4.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target >2012-11-08 06:04:06,569 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-95) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv4.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target >2012-11-08 06:04:08,587 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-97) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv4.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target >2012-11-08 06:04:10,603 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-1) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv4.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target >2012-11-08 06:04:12,617 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-5) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv4.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target >2012-11-08 06:04:14,637 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-8) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv4.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target >2012-11-08 06:04:16,653 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-12) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv4.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target >2012-11-08 06:04:18,676 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-16) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv4.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target >2012-11-08 06:04:20,691 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-19) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv4.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target >2012-11-08 06:04:22,708 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-21) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv4.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target >2012-11-08 06:04:24,723 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-25) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv4.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target >2012-11-08 06:04:26,738 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-28) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv4.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target >2012-11-08 06:04:28,759 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-33) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv4.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target >2012-11-08 06:04:30,776 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-36) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv4.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target >2012-11-08 06:04:32,796 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-40) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv4.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target >2012-11-08 06:04:34,813 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-42) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv4.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target >2012-11-08 06:04:36,831 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-46) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv4.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target >2012-11-08 06:04:38,848 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-50) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv4.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target >2012-11-08 06:04:40,877 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-53) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv4.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target >2012-11-08 06:04:42,894 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-58) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv4.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target >2012-11-08 06:04:42,930 INFO [org.ovirt.engine.core.bll.storage.FenceVdsManualyCommand] (ajp-/127.0.0.1:8702-8) Lock Acquired to object EngineLock [exclusiveLocks= key: 96739be4-2990-11e2-8577-52540035ed48 value: VDS_FENCE >, sharedLocks= ] >2012-11-08 06:04:42,931 INFO [org.ovirt.engine.core.bll.storage.FenceVdsManualyCommand] (pool-4-thread-49) [52d371d9] Running command: FenceVdsManualyCommand internal: false. Entities affected : ID: 96739be4-2990-11e2-8577-52540035ed48 Type: VDS >2012-11-08 06:04:42,935 INFO [org.ovirt.engine.core.bll.ClearNonResponsiveVdsVmsCommand] (pool-4-thread-49) [5e1bf8d6] Running command: ClearNonResponsiveVdsVmsCommand internal: true. Entities affected : ID: 96739be4-2990-11e2-8577-52540035ed48 Type: VDS >2012-11-08 06:04:42,937 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsVMsClearedVDSCommand] (pool-4-thread-49) [5e1bf8d6] START, UpdateVdsVMsClearedVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48), log id: 3ebcb90f >2012-11-08 06:04:42,938 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsVMsClearedVDSCommand] (pool-4-thread-49) [5e1bf8d6] FINISH, UpdateVdsVMsClearedVDSCommand, log id: 3ebcb90f >2012-11-08 06:04:42,948 INFO [org.ovirt.engine.core.bll.storage.FenceVdsManualyCommand] (pool-4-thread-49) [5e1bf8d6] Lock freed to object EngineLock [exclusiveLocks= key: 96739be4-2990-11e2-8577-52540035ed48 value: VDS_FENCE >, sharedLocks= ] >2012-11-08 06:04:44,914 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-60) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv4.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target >2012-11-08 06:04:46,932 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-62) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv4.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target >2012-11-08 06:04:47,474 INFO [org.ovirt.engine.core.bll.storage.FenceVdsManualyCommand] (ajp-/127.0.0.1:8702-2) Lock Acquired to object EngineLock [exclusiveLocks= key: d2bd32d4-2992-11e2-ab59-52540035ed48 value: VDS_FENCE >, sharedLocks= ] >2012-11-08 06:04:47,474 INFO [org.ovirt.engine.core.bll.storage.FenceVdsManualyCommand] (pool-4-thread-49) [2d87b5f7] Running command: FenceVdsManualyCommand internal: false. Entities affected : ID: d2bd32d4-2992-11e2-ab59-52540035ed48 Type: VDS >2012-11-08 06:04:47,478 INFO [org.ovirt.engine.core.bll.ClearNonResponsiveVdsVmsCommand] (pool-4-thread-49) [29706815] Running command: ClearNonResponsiveVdsVmsCommand internal: true. Entities affected : ID: d2bd32d4-2992-11e2-ab59-52540035ed48 Type: VDS >2012-11-08 06:04:47,480 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsVMsClearedVDSCommand] (pool-4-thread-49) [29706815] START, UpdateVdsVMsClearedVDSCommand(HostName = rhs-gp-srv4.lab.eng.blr.redhat.com, HostId = d2bd32d4-2992-11e2-ab59-52540035ed48), log id: 34da9d5a >2012-11-08 06:04:47,481 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsVMsClearedVDSCommand] (pool-4-thread-49) [29706815] FINISH, UpdateVdsVMsClearedVDSCommand, log id: 34da9d5a >2012-11-08 06:04:47,486 INFO [org.ovirt.engine.core.bll.storage.FenceVdsManualyCommand] (pool-4-thread-49) [29706815] Lock freed to object EngineLock [exclusiveLocks= key: d2bd32d4-2992-11e2-ab59-52540035ed48 value: VDS_FENCE >, sharedLocks= ] >2012-11-08 06:04:48,945 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-66) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv4.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target >2012-11-08 06:04:50,967 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-70) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv4.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target >2012-11-08 06:04:52,988 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-73) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv4.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target >2012-11-08 06:04:55,002 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-75) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv4.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target >2012-11-08 06:04:57,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (ajp-/127.0.0.1:8702-12) Lock Acquired to object EngineLock [exclusiveLocks= key: 96739be4-2990-11e2-8577-52540035ed48 value: VDS >, sharedLocks= ] >2012-11-08 06:04:57,018 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-82) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv4.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target >2012-11-08 06:04:57,035 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (pool-4-thread-49) [517a07db] Running command: ActivateVdsCommand internal: false. Entities affected : ID: 96739be4-2990-11e2-8577-52540035ed48 Type: VDS >2012-11-08 06:04:57,042 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-49) [517a07db] START, SetVdsStatusVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, status=Unassigned, nonOperationalReason=NONE), log id: 5211344 >2012-11-08 06:04:57,044 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-49) [517a07db] FINISH, SetVdsStatusVDSCommand, log id: 5211344 >2012-11-08 06:04:57,047 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (pool-4-thread-49) [517a07db] START, ActivateVdsVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48), log id: 4d686fea >2012-11-08 06:04:57,063 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (pool-4-thread-49) [517a07db] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv11.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target >2012-11-08 06:04:57,063 ERROR [org.ovirt.engine.core.vdsbroker.VdsManager] (pool-4-thread-49) [517a07db] VDS::handleNetworkException Server failed to respond, vds_id = 96739be4-2990-11e2-8577-52540035ed48, vds_name = rhs-gp-srv11.lab.eng.blr.redhat.com, error = VDSNetworkException: >2012-11-08 06:04:57,078 INFO [org.ovirt.engine.core.bll.VdsEventListener] (pool-4-thread-50) [517a07db] ResourceManager::vdsNotResponding entered for Host 96739be4-2990-11e2-8577-52540035ed48, 10.70.36.15 >2012-11-08 06:04:57,082 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (pool-4-thread-49) [517a07db] ResourceManager::activateVds - failed to get VDS = 96739be4-2990-11e2-8577-52540035ed48 capabilities with error: VDSNetworkException: . >2012-11-08 06:04:57,082 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (pool-4-thread-49) [517a07db] ResourceManager::activateVds - failed to activate VDS = 96739be4-2990-11e2-8577-52540035ed48 >2012-11-08 06:04:57,086 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (pool-4-thread-49) [9a3f03] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 96739be4-2990-11e2-8577-52540035ed48 Type: VDS >2012-11-08 06:04:57,087 WARN [org.ovirt.engine.core.bll.VdsNotRespondingTreatmentCommand] (pool-4-thread-50) [f08a676] CanDoAction of action VdsNotRespondingTreatment failed. Reasons:VDS_FENCING_DISABLED >2012-11-08 06:04:57,096 WARN [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (pool-4-thread-49) [6160c6b6] CanDoAction of action HandleVdsVersion failed. Reasons:VDS_CANNOT_CHECK_VERSION_HOST_NON_RESPONSIVE >2012-11-08 06:04:57,097 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (pool-4-thread-49) [6160c6b6] FINISH, ActivateVdsVDSCommand, log id: 4d686fea >2012-11-08 06:04:57,116 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (pool-4-thread-49) [6160c6b6] Lock freed to object EngineLock [exclusiveLocks= key: 96739be4-2990-11e2-8577-52540035ed48 value: VDS >, sharedLocks= ] >2012-11-08 06:04:58,042 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-84) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv11.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target >2012-11-08 06:04:59,036 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-87) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv4.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target >2012-11-08 06:05:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-88) Autorecovering hosts is disabled, skipping >2012-11-08 06:05:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-88) Autorecovering storage domains is disabled, skipping >2012-11-08 06:05:00,060 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-89) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv11.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target >2012-11-08 06:05:01,049 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-91) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv4.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target >2012-11-08 06:05:02,087 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-92) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv11.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target >2012-11-08 06:05:03,063 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-95) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv4.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target >2012-11-08 06:05:03,617 INFO [org.ovirt.engine.core.bll.MaintananceNumberOfVdssCommand] (pool-4-thread-49) [7a7f3a20] Running command: MaintananceNumberOfVdssCommand internal: false. Entities affected : ID: d2bd32d4-2992-11e2-ab59-52540035ed48 Type: VDS >2012-11-08 06:05:03,632 INFO [org.ovirt.engine.core.bll.MaintananceVdsCommand] (pool-4-thread-49) [7a7f3a20] Running command: MaintananceVdsCommand internal: true. Entities affected : ID: d2bd32d4-2992-11e2-ab59-52540035ed48 Type: VDS >2012-11-08 06:05:03,633 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-49) [7a7f3a20] START, SetVdsStatusVDSCommand(HostName = rhs-gp-srv4.lab.eng.blr.redhat.com, HostId = d2bd32d4-2992-11e2-ab59-52540035ed48, status=Maintenance, nonOperationalReason=NONE), log id: 3300553 >2012-11-08 06:05:03,634 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-49) [7a7f3a20] FINISH, SetVdsStatusVDSCommand, log id: 3300553 >2012-11-08 06:05:04,102 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-94) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv11.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target >2012-11-08 06:05:06,116 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-98) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv11.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target >2012-11-08 06:05:08,131 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-2) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv11.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target >2012-11-08 06:05:10,150 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-7) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv11.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target >2012-11-08 06:05:12,167 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-6) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv11.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target >2012-11-08 06:05:14,184 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-13) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv11.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target >2012-11-08 06:05:15,717 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (ajp-/127.0.0.1:8702-8) Lock Acquired to object EngineLock [exclusiveLocks= key: d2bd32d4-2992-11e2-ab59-52540035ed48 value: VDS >, sharedLocks= ] >2012-11-08 06:05:15,739 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (pool-4-thread-49) [71e04fa4] Running command: ActivateVdsCommand internal: false. Entities affected : ID: d2bd32d4-2992-11e2-ab59-52540035ed48 Type: VDS >2012-11-08 06:05:15,753 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-49) [71e04fa4] START, SetVdsStatusVDSCommand(HostName = rhs-gp-srv4.lab.eng.blr.redhat.com, HostId = d2bd32d4-2992-11e2-ab59-52540035ed48, status=Unassigned, nonOperationalReason=NONE), log id: 30c9b8dc >2012-11-08 06:05:15,756 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-49) [71e04fa4] FINISH, SetVdsStatusVDSCommand, log id: 30c9b8dc >2012-11-08 06:05:15,759 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (pool-4-thread-49) [71e04fa4] START, ActivateVdsVDSCommand(HostName = rhs-gp-srv4.lab.eng.blr.redhat.com, HostId = d2bd32d4-2992-11e2-ab59-52540035ed48), log id: 45c71708 >2012-11-08 06:05:15,768 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (pool-4-thread-49) [71e04fa4] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv4.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target >2012-11-08 06:05:15,768 ERROR [org.ovirt.engine.core.vdsbroker.VdsManager] (pool-4-thread-49) [71e04fa4] VDS::handleNetworkException Server failed to respond, vds_id = d2bd32d4-2992-11e2-ab59-52540035ed48, vds_name = rhs-gp-srv4.lab.eng.blr.redhat.com, error = VDSNetworkException: >2012-11-08 06:05:15,776 INFO [org.ovirt.engine.core.bll.VdsEventListener] (pool-4-thread-50) [71e04fa4] ResourceManager::vdsNotResponding entered for Host d2bd32d4-2992-11e2-ab59-52540035ed48, 10.70.36.8 >2012-11-08 06:05:15,783 WARN [org.ovirt.engine.core.bll.VdsNotRespondingTreatmentCommand] (pool-4-thread-50) [21d8ccf0] CanDoAction of action VdsNotRespondingTreatment failed. Reasons:VDS_FENCING_DISABLED >2012-11-08 06:05:15,784 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (pool-4-thread-49) [71e04fa4] ResourceManager::activateVds - failed to get VDS = d2bd32d4-2992-11e2-ab59-52540035ed48 capabilities with error: VDSNetworkException: . >2012-11-08 06:05:15,784 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (pool-4-thread-49) [71e04fa4] ResourceManager::activateVds - failed to activate VDS = d2bd32d4-2992-11e2-ab59-52540035ed48 >2012-11-08 06:05:15,788 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (pool-4-thread-49) [356b77af] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: d2bd32d4-2992-11e2-ab59-52540035ed48 Type: VDS >2012-11-08 06:05:15,790 WARN [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (pool-4-thread-49) [763c1864] CanDoAction of action HandleVdsVersion failed. Reasons:VDS_CANNOT_CHECK_VERSION_HOST_NON_RESPONSIVE >2012-11-08 06:05:15,790 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (pool-4-thread-49) [763c1864] FINISH, ActivateVdsVDSCommand, log id: 45c71708 >2012-11-08 06:05:15,807 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (pool-4-thread-49) [763c1864] Lock freed to object EngineLock [exclusiveLocks= key: d2bd32d4-2992-11e2-ab59-52540035ed48 value: VDS >, sharedLocks= ] >2012-11-08 06:05:16,199 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-17) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv11.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target >2012-11-08 06:05:17,089 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-19) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv4.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target >2012-11-08 06:05:18,216 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-20) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv11.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target >2012-11-08 06:05:18,897 INFO [org.ovirt.engine.core.bll.MaintananceNumberOfVdssCommand] (pool-4-thread-49) [6495791f] Running command: MaintananceNumberOfVdssCommand internal: false. Entities affected : ID: 96739be4-2990-11e2-8577-52540035ed48 Type: VDS >2012-11-08 06:05:18,926 INFO [org.ovirt.engine.core.bll.MaintananceVdsCommand] (pool-4-thread-49) [6495791f] Running command: MaintananceVdsCommand internal: true. Entities affected : ID: 96739be4-2990-11e2-8577-52540035ed48 Type: VDS >2012-11-08 06:05:18,928 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-49) [6495791f] START, SetVdsStatusVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, status=Maintenance, nonOperationalReason=NONE), log id: 3a9b723 >2012-11-08 06:05:18,930 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-49) [6495791f] FINISH, SetVdsStatusVDSCommand, log id: 3a9b723 >2012-11-08 06:05:19,102 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-23) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv4.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target >2012-11-08 06:05:21,113 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-26) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv4.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target >2012-11-08 06:05:23,135 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-9) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv4.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target >2012-11-08 06:05:25,151 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-32) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv4.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target >2012-11-08 06:05:25,222 INFO [org.ovirt.engine.core.bll.MaintananceNumberOfVdssCommand] (pool-4-thread-49) [1151cde3] Running command: MaintananceNumberOfVdssCommand internal: false. Entities affected : ID: d2bd32d4-2992-11e2-ab59-52540035ed48 Type: VDS >2012-11-08 06:05:25,257 INFO [org.ovirt.engine.core.bll.MaintananceVdsCommand] (pool-4-thread-49) [1151cde3] Running command: MaintananceVdsCommand internal: true. Entities affected : ID: d2bd32d4-2992-11e2-ab59-52540035ed48 Type: VDS >2012-11-08 06:05:25,259 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-49) [1151cde3] START, SetVdsStatusVDSCommand(HostName = rhs-gp-srv4.lab.eng.blr.redhat.com, HostId = d2bd32d4-2992-11e2-ab59-52540035ed48, status=Maintenance, nonOperationalReason=NONE), log id: 9d02519 >2012-11-08 06:05:25,261 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-49) [1151cde3] FINISH, SetVdsStatusVDSCommand, log id: 9d02519 >2012-11-08 06:05:33,107 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (ajp-/127.0.0.1:8702-6) Lock Acquired to object EngineLock [exclusiveLocks= key: 96739be4-2990-11e2-8577-52540035ed48 value: VDS >, sharedLocks= ] >2012-11-08 06:05:33,125 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (pool-4-thread-49) [139930f2] Running command: ActivateVdsCommand internal: false. Entities affected : ID: 96739be4-2990-11e2-8577-52540035ed48 Type: VDS >2012-11-08 06:05:33,143 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-49) [139930f2] START, SetVdsStatusVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, status=Unassigned, nonOperationalReason=NONE), log id: 40ad776d >2012-11-08 06:05:33,146 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-49) [139930f2] FINISH, SetVdsStatusVDSCommand, log id: 40ad776d >2012-11-08 06:05:33,148 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (pool-4-thread-49) [139930f2] START, ActivateVdsVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48), log id: 4b9aa7ea >2012-11-08 06:05:33,155 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (pool-4-thread-49) [139930f2] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv11.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target >2012-11-08 06:05:33,155 ERROR [org.ovirt.engine.core.vdsbroker.VdsManager] (pool-4-thread-49) [139930f2] VDS::handleNetworkException Server failed to respond, vds_id = 96739be4-2990-11e2-8577-52540035ed48, vds_name = rhs-gp-srv11.lab.eng.blr.redhat.com, error = VDSNetworkException: >2012-11-08 06:05:33,171 INFO [org.ovirt.engine.core.bll.VdsEventListener] (pool-4-thread-50) [139930f2] ResourceManager::vdsNotResponding entered for Host 96739be4-2990-11e2-8577-52540035ed48, 10.70.36.15 >2012-11-08 06:05:33,175 WARN [org.ovirt.engine.core.bll.VdsNotRespondingTreatmentCommand] (pool-4-thread-50) [600bd0a7] CanDoAction of action VdsNotRespondingTreatment failed. Reasons:VDS_FENCING_DISABLED >2012-11-08 06:05:33,177 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (pool-4-thread-49) [139930f2] ResourceManager::activateVds - failed to get VDS = 96739be4-2990-11e2-8577-52540035ed48 capabilities with error: VDSNetworkException: . >2012-11-08 06:05:33,177 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (pool-4-thread-49) [139930f2] ResourceManager::activateVds - failed to activate VDS = 96739be4-2990-11e2-8577-52540035ed48 >2012-11-08 06:05:33,187 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (pool-4-thread-49) [1237287b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 96739be4-2990-11e2-8577-52540035ed48 Type: VDS >2012-11-08 06:05:33,189 WARN [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (pool-4-thread-49) [40bb7ea] CanDoAction of action HandleVdsVersion failed. Reasons:VDS_CANNOT_CHECK_VERSION_HOST_NON_RESPONSIVE >2012-11-08 06:05:33,189 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (pool-4-thread-49) [40bb7ea] FINISH, ActivateVdsVDSCommand, log id: 4b9aa7ea >2012-11-08 06:05:33,204 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (pool-4-thread-49) [40bb7ea] Lock freed to object EngineLock [exclusiveLocks= key: 96739be4-2990-11e2-8577-52540035ed48 value: VDS >, sharedLocks= ] >2012-11-08 06:05:34,246 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-47) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv11.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target >2012-11-08 06:05:36,261 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-50) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv11.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target >2012-11-08 06:05:38,136 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (ajp-/127.0.0.1:8702-6) Lock Acquired to object EngineLock [exclusiveLocks= key: d2bd32d4-2992-11e2-ab59-52540035ed48 value: VDS >, sharedLocks= ] >2012-11-08 06:05:38,152 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (pool-4-thread-49) [79090585] Running command: ActivateVdsCommand internal: false. Entities affected : ID: d2bd32d4-2992-11e2-ab59-52540035ed48 Type: VDS >2012-11-08 06:05:38,166 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-49) [79090585] START, SetVdsStatusVDSCommand(HostName = rhs-gp-srv4.lab.eng.blr.redhat.com, HostId = d2bd32d4-2992-11e2-ab59-52540035ed48, status=Unassigned, nonOperationalReason=NONE), log id: 217950e1 >2012-11-08 06:05:38,168 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-49) [79090585] FINISH, SetVdsStatusVDSCommand, log id: 217950e1 >2012-11-08 06:05:38,170 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (pool-4-thread-49) [79090585] START, ActivateVdsVDSCommand(HostName = rhs-gp-srv4.lab.eng.blr.redhat.com, HostId = d2bd32d4-2992-11e2-ab59-52540035ed48), log id: 7d95db04 >2012-11-08 06:05:38,180 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (pool-4-thread-49) [79090585] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv4.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target >2012-11-08 06:05:38,180 ERROR [org.ovirt.engine.core.vdsbroker.VdsManager] (pool-4-thread-49) [79090585] VDS::handleNetworkException Server failed to respond, vds_id = d2bd32d4-2992-11e2-ab59-52540035ed48, vds_name = rhs-gp-srv4.lab.eng.blr.redhat.com, error = VDSNetworkException: >2012-11-08 06:05:38,193 INFO [org.ovirt.engine.core.bll.VdsEventListener] (pool-4-thread-50) [79090585] ResourceManager::vdsNotResponding entered for Host d2bd32d4-2992-11e2-ab59-52540035ed48, 10.70.36.8 >2012-11-08 06:05:38,198 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (pool-4-thread-49) [79090585] ResourceManager::activateVds - failed to get VDS = d2bd32d4-2992-11e2-ab59-52540035ed48 capabilities with error: VDSNetworkException: . >2012-11-08 06:05:38,198 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (pool-4-thread-49) [79090585] ResourceManager::activateVds - failed to activate VDS = d2bd32d4-2992-11e2-ab59-52540035ed48 >2012-11-08 06:05:38,201 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (pool-4-thread-49) [5ec2e430] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: d2bd32d4-2992-11e2-ab59-52540035ed48 Type: VDS >2012-11-08 06:05:38,202 WARN [org.ovirt.engine.core.bll.VdsNotRespondingTreatmentCommand] (pool-4-thread-50) [38f4be0b] CanDoAction of action VdsNotRespondingTreatment failed. Reasons:VDS_FENCING_DISABLED >2012-11-08 06:05:38,204 WARN [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (pool-4-thread-49) [1ace768d] CanDoAction of action HandleVdsVersion failed. Reasons:VDS_CANNOT_CHECK_VERSION_HOST_NON_RESPONSIVE >2012-11-08 06:05:38,204 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (pool-4-thread-49) [1ace768d] FINISH, ActivateVdsVDSCommand, log id: 7d95db04 >2012-11-08 06:05:38,216 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (pool-4-thread-49) [1ace768d] Lock freed to object EngineLock [exclusiveLocks= key: d2bd32d4-2992-11e2-ab59-52540035ed48 value: VDS >, sharedLocks= ] >2012-11-08 06:05:38,281 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-54) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv11.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target >2012-11-08 06:05:39,178 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-57) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv4.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target >2012-11-08 06:05:40,297 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-56) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv11.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target >2012-11-08 06:05:41,194 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-59) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv4.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target >2012-11-08 06:05:42,311 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-61) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv11.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target >2012-11-08 06:05:43,210 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-62) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv4.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target >2012-11-08 06:05:44,326 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-64) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv11.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target >2012-11-08 06:05:44,881 INFO [org.ovirt.engine.core.bll.MaintananceNumberOfVdssCommand] (pool-4-thread-49) [1fb6c1c6] Running command: MaintananceNumberOfVdssCommand internal: false. Entities affected : ID: d2bd32d4-2992-11e2-ab59-52540035ed48 Type: VDS >2012-11-08 06:05:44,895 INFO [org.ovirt.engine.core.bll.MaintananceVdsCommand] (pool-4-thread-49) [1fb6c1c6] Running command: MaintananceVdsCommand internal: true. Entities affected : ID: d2bd32d4-2992-11e2-ab59-52540035ed48 Type: VDS >2012-11-08 06:05:44,898 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-49) [1fb6c1c6] START, SetVdsStatusVDSCommand(HostName = rhs-gp-srv4.lab.eng.blr.redhat.com, HostId = d2bd32d4-2992-11e2-ab59-52540035ed48, status=Maintenance, nonOperationalReason=NONE), log id: 156bdb02 >2012-11-08 06:05:44,900 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-49) [1fb6c1c6] FINISH, SetVdsStatusVDSCommand, log id: 156bdb02 >2012-11-08 06:05:46,343 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-66) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv11.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target >2012-11-08 06:05:48,359 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-71) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv11.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, SunCertPathBuilderException: unable to find valid certification path to requested target >2012-11-08 06:05:48,721 INFO [org.ovirt.engine.core.bll.MaintananceNumberOfVdssCommand] (pool-4-thread-49) [6bfa6c51] Running command: MaintananceNumberOfVdssCommand internal: false. Entities affected : ID: 96739be4-2990-11e2-8577-52540035ed48 Type: VDS >2012-11-08 06:05:48,739 INFO [org.ovirt.engine.core.bll.MaintananceVdsCommand] (pool-4-thread-49) [6bfa6c51] Running command: MaintananceVdsCommand internal: true. Entities affected : ID: 96739be4-2990-11e2-8577-52540035ed48 Type: VDS >2012-11-08 06:05:48,742 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-49) [6bfa6c51] START, SetVdsStatusVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, status=Maintenance, nonOperationalReason=NONE), log id: 60a55171 >2012-11-08 06:05:48,744 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-49) [6bfa6c51] FINISH, SetVdsStatusVDSCommand, log id: 60a55171 >2012-11-08 06:05:58,300 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp-/127.0.0.1:8702-6) [6f2e9c96] Running command: UpdateVdsCommand internal: false. Entities affected : ID: 96739be4-2990-11e2-8577-52540035ed48 Type: VDS >2012-11-08 06:05:58,316 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp-/127.0.0.1:8702-6) [6f2e9c96] START, SetVdsStatusVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, status=Installing, nonOperationalReason=NONE), log id: 64b0f048 >2012-11-08 06:05:58,318 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp-/127.0.0.1:8702-6) [6f2e9c96] FINISH, SetVdsStatusVDSCommand, log id: 64b0f048 >2012-11-08 06:05:58,321 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-4-thread-49) [6f7c3581] Running command: InstallVdsCommand internal: true. Entities affected : ID: 96739be4-2990-11e2-8577-52540035ed48 Type: VDS >2012-11-08 06:05:58,321 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-4-thread-49) [6f7c3581] Before Installation pool-4-thread-49 >2012-11-08 06:05:58,322 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-49) [6f7c3581] Installation of 10.70.36.15. Executing installation stage. (Stage: Starting Host installation) >2012-11-08 06:05:58,322 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-49) [6f7c3581] Installation of 10.70.36.15. Executing installation stage. (Stage: Connecting to Host) >2012-11-08 06:05:58,383 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-49) [6f7c3581] Installation of 10.70.36.15. Received message: <BSTRAP component='RHEV_INSTALL' status='OK' message='Connected to Host 10.70.36.15 with SSH key fingerprint: cc:0f:a3:d7:15:b2:45:74:3a:6b:c6:2b:4b:f5:0d:0d'/>. FYI. (Stage: Connecting to Host) >2012-11-08 06:05:58,423 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-49) [6f7c3581] Installation of 10.70.36.15. Successfully connected to server ssh. (Stage: Connecting to Host) >2012-11-08 06:05:58,423 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-49) [6f7c3581] Installation of 10.70.36.15. Executing installation stage. (Stage: Get the unique vds id) >2012-11-08 06:05:58,423 INFO [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-4-thread-49) SSH execute 10.70.36.15 'IDFILE=/etc/vdsm/vdsm.id; if [ -r "${IDFILE}" ]; then cat "${IDFILE}"; else UUID="$(dmidecode -s system-uuid 2> /dev/null | sed -e 's/.*Not.*//' )"; if [ -z "${UUID}" ]; then UUID="$(uuidgen 2> /dev/null)" && mkdir -p "$(dirname "${IDFILE}")" && echo "${UUID}" > "${IDFILE}" && chmod 0644 "${IDFILE}"; fi; [ -n "${UUID}" ] && echo "${UUID}"; fi' >2012-11-08 06:05:58,536 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-38) Installation of 10.70.36.15. Received message: 00000000-0000-0000-0000-0025902F7254. FYI. (Stage: Get the unique vds id) >2012-11-08 06:05:58,538 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-38) Installation of 10.70.36.15. Assigning unique id 00000000-0000-0000-0000-0025902F7254 to Host. (Stage: Get the unique vds id) >2012-11-08 06:05:58,540 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-49) [6f7c3581] Installation of 10.70.36.15. Executing installation stage. (Stage: Upload Installation script to Host) >2012-11-08 06:05:58,541 INFO [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-4-thread-49) SSH execute 10.70.36.15 'cat > '/tmp/ovirt-id_rsa_014162ae-00de-44fa-8855-4bbafa8d6378'' >2012-11-08 06:05:58,626 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-49) [6f7c3581] Installation of 10.70.36.15. Executing installation stage. (Stage: Running first installation script on Host) >2012-11-08 06:05:58,626 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-49) [6f7c3581] Installation of 10.70.36.15. Sending SSH Command umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar -C "${MYTMP}" --no-same-permissions -o -x && "${MYTMP}"/setup -c 'ssl=true;management_port=54321' -O 'lab.eng.blr.redhat.com' -t 2012-11-08T11:05:58 -S /tmp/ovirt-id_rsa_014162ae-00de-44fa-8855-4bbafa8d6378 -p 80 -b -B rhevm http://guido.lab.eng.blr.redhat.com:80/Components/vds/ http://guido.lab.eng.blr.redhat.com:80/Components/vds/ 10.70.36.15 014162ae-00de-44fa-8855-4bbafa8d6378 False < /var/cache/ovirt-engine/vdsm-bootstrap-2.tar. (Stage: Running first installation script on Host) >2012-11-08 06:05:58,626 INFO [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-4-thread-49) SSH execute 10.70.36.15 'umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar -C "${MYTMP}" --no-same-permissions -o -x && "${MYTMP}"/setup -c 'ssl=true;management_port=54321' -O 'lab.eng.blr.redhat.com' -t 2012-11-08T11:05:58 -S /tmp/ovirt-id_rsa_014162ae-00de-44fa-8855-4bbafa8d6378 -p 80 -b -B rhevm http://guido.lab.eng.blr.redhat.com:80/Components/vds/ http://guido.lab.eng.blr.redhat.com:80/Components/vds/ 10.70.36.15 014162ae-00de-44fa-8855-4bbafa8d6378 False' >2012-11-08 06:05:58,731 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-38) Installation of 10.70.36.15. Received message: <BSTRAP component='INSTALLER' status='OK' message='Test platform succeeded'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 06:05:59,855 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-38) Installation of 10.70.36.15. Received message: <BSTRAP component='SET_SYSTEM_TIME' status='OK' message='setSystemTime ended successfully'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 06:06:02,981 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-38) Installation of 10.70.36.15. Received message: <BSTRAP component='RHN_REGISTRATION' status='OK' message='Host properly registered with RHN/Satellite.'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 06:06:03,128 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-38) Installation of 10.70.36.15. Received message: <BSTRAP component='VDSM_MAJOR_VER' status='OK' message='Available VDSM matches requirements'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 06:06:03,133 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-38) Installation of 10.70.36.15. Received message: <BSTRAP component='VT_SVM' status='OK' processor='Intel' message='Server supports virtualization'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 06:06:03,138 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-38) Installation of 10.70.36.15. Received message: <BSTRAP component='OS' status='OK' type='RHEL6' message='Supported platform version'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 06:06:03,142 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-38) Installation of 10.70.36.15. Received message: <BSTRAP component='KERNEL' status='OK' version='279' message='Supported kernel version: 279'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 06:06:03,236 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-38) Installation of 10.70.36.15. Received message: <BSTRAP component='REQ PACKAGES' status='OK' result='SDL.x86_64' message='SDL-1.2.14-3.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 06:06:03,255 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-38) Installation of 10.70.36.15. Received message: <BSTRAP component='REQ PACKAGES' status='OK' result='bridge-utils.x86_64' message='bridge-utils-1.2-9.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 06:06:03,273 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-38) Installation of 10.70.36.15. Received message: <BSTRAP component='REQ PACKAGES' status='OK' result='mesa-libGLU.x86_64' message='mesa-libGLU-7.11-5.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 06:06:03,296 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-38) Installation of 10.70.36.15. Received message: <BSTRAP component='REQ PACKAGES' status='OK' result='openssl.x86_64' message='openssl-1.0.0-25.el6_3.1.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 06:06:03,319 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-38) Installation of 10.70.36.15. Received message: <BSTRAP component='REQ PACKAGES' status='OK' result='m2crypto.x86_64' message='m2crypto-0.20.2-9.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 06:06:03,334 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-38) Installation of 10.70.36.15. Received message: <BSTRAP component='REQ PACKAGES' status='OK' result='rsync.x86_64' message='rsync-3.0.6-9.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 06:06:03,347 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-38) Installation of 10.70.36.15. Received message: <BSTRAP component='VDS PACKAGES' status='WARN' result='qemu-kvm' message='package qemu-kvm is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 06:06:03,360 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-38) Installation of 10.70.36.15. Received message: <BSTRAP component='VDS PACKAGES' status='WARN' result='qemu-kvm-tools' message='package qemu-kvm-tools is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 06:06:03,378 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-38) Installation of 10.70.36.15. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='(~vdsm~, ~vdsm-cli~)' message='vdsm-4.9.6-41.0.el6_3.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 06:06:03,393 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-38) Installation of 10.70.36.15. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='libjpeg' message='libjpeg-6b-46.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 06:06:03,408 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-38) Installation of 10.70.36.15. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='spice-server' message='spice-server-0.10.1-10.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 06:06:03,427 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-38) Installation of 10.70.36.15. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='pixman' message='pixman-0.18.4-1.el6_0.1.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 06:06:03,450 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-38) Installation of 10.70.36.15. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='seabios' message='seabios-0.6.1.2-19.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 06:06:03,463 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-38) Installation of 10.70.36.15. Received message: <BSTRAP component='VDS PACKAGES' status='WARN' result='qemu-img' message='package qemu-img is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 06:06:03,485 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-38) Installation of 10.70.36.15. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='fence-agents' message='fence-agents-3.1.5-17.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 06:06:03,514 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-38) Installation of 10.70.36.15. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='libselinux-python' message='libselinux-python-2.0.94-5.3.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 06:06:03,528 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-38) Installation of 10.70.36.15. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='sanlock' message='sanlock-2.3-4.el6_3.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 06:06:03,550 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-38) Installation of 10.70.36.15. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='sanlock-python' message='sanlock-python-2.3-4.el6_3.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 06:06:06,310 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp-/127.0.0.1:8702-1) [1d1fb618] Running command: UpdateVdsCommand internal: false. Entities affected : ID: d2bd32d4-2992-11e2-ab59-52540035ed48 Type: VDS >2012-11-08 06:06:06,320 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp-/127.0.0.1:8702-1) [1d1fb618] START, SetVdsStatusVDSCommand(HostName = rhs-gp-srv4.lab.eng.blr.redhat.com, HostId = d2bd32d4-2992-11e2-ab59-52540035ed48, status=Installing, nonOperationalReason=NONE), log id: 3a33660f >2012-11-08 06:06:06,321 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp-/127.0.0.1:8702-1) [1d1fb618] FINISH, SetVdsStatusVDSCommand, log id: 3a33660f >2012-11-08 06:06:06,327 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-4-thread-50) [26b62dee] Running command: InstallVdsCommand internal: true. Entities affected : ID: d2bd32d4-2992-11e2-ab59-52540035ed48 Type: VDS >2012-11-08 06:06:06,328 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-4-thread-50) [26b62dee] Before Installation pool-4-thread-50 >2012-11-08 06:06:06,328 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-50) [26b62dee] Installation of 10.70.36.8. Executing installation stage. (Stage: Starting Host installation) >2012-11-08 06:06:06,328 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-50) [26b62dee] Installation of 10.70.36.8. Executing installation stage. (Stage: Connecting to Host) >2012-11-08 06:06:06,402 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-50) [26b62dee] Installation of 10.70.36.8. Received message: <BSTRAP component='RHEV_INSTALL' status='OK' message='Connected to Host 10.70.36.8 with SSH key fingerprint: be:b4:2c:77:3c:6e:60:f7:35:69:fb:29:52:29:15:e1'/>. FYI. (Stage: Connecting to Host) >2012-11-08 06:06:06,433 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-50) [26b62dee] Installation of 10.70.36.8. Successfully connected to server ssh. (Stage: Connecting to Host) >2012-11-08 06:06:06,433 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-50) [26b62dee] Installation of 10.70.36.8. Executing installation stage. (Stage: Get the unique vds id) >2012-11-08 06:06:06,433 INFO [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-4-thread-50) SSH execute 10.70.36.8 'IDFILE=/etc/vdsm/vdsm.id; if [ -r "${IDFILE}" ]; then cat "${IDFILE}"; else UUID="$(dmidecode -s system-uuid 2> /dev/null | sed -e 's/.*Not.*//' )"; if [ -z "${UUID}" ]; then UUID="$(uuidgen 2> /dev/null)" && mkdir -p "$(dirname "${IDFILE}")" && echo "${UUID}" > "${IDFILE}" && chmod 0644 "${IDFILE}"; fi; [ -n "${UUID}" ] && echo "${UUID}"; fi' >2012-11-08 06:06:06,548 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-42) Installation of 10.70.36.8. Received message: 00000000-0000-0000-0000-0025902F8CA8. FYI. (Stage: Get the unique vds id) >2012-11-08 06:06:06,549 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-42) Installation of 10.70.36.8. Assigning unique id 00000000-0000-0000-0000-0025902F8CA8 to Host. (Stage: Get the unique vds id) >2012-11-08 06:06:06,552 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-50) [26b62dee] Installation of 10.70.36.8. Executing installation stage. (Stage: Upload Installation script to Host) >2012-11-08 06:06:06,552 INFO [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-4-thread-50) SSH execute 10.70.36.8 'cat > '/tmp/ovirt-id_rsa_bb4b9ad9-cf23-414f-9c82-fd3b97d6f24b'' >2012-11-08 06:06:06,637 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-50) [26b62dee] Installation of 10.70.36.8. Executing installation stage. (Stage: Running first installation script on Host) >2012-11-08 06:06:06,637 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-50) [26b62dee] Installation of 10.70.36.8. Sending SSH Command umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar -C "${MYTMP}" --no-same-permissions -o -x && "${MYTMP}"/setup -c 'ssl=true;management_port=54321' -O 'lab.eng.blr.redhat.com' -t 2012-11-08T11:06:06 -S /tmp/ovirt-id_rsa_bb4b9ad9-cf23-414f-9c82-fd3b97d6f24b -p 80 -b -B rhevm http://guido.lab.eng.blr.redhat.com:80/Components/vds/ http://guido.lab.eng.blr.redhat.com:80/Components/vds/ 10.70.36.8 bb4b9ad9-cf23-414f-9c82-fd3b97d6f24b False < /var/cache/ovirt-engine/vdsm-bootstrap-2.tar. (Stage: Running first installation script on Host) >2012-11-08 06:06:06,637 INFO [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-4-thread-50) SSH execute 10.70.36.8 'umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar -C "${MYTMP}" --no-same-permissions -o -x && "${MYTMP}"/setup -c 'ssl=true;management_port=54321' -O 'lab.eng.blr.redhat.com' -t 2012-11-08T11:06:06 -S /tmp/ovirt-id_rsa_bb4b9ad9-cf23-414f-9c82-fd3b97d6f24b -p 80 -b -B rhevm http://guido.lab.eng.blr.redhat.com:80/Components/vds/ http://guido.lab.eng.blr.redhat.com:80/Components/vds/ 10.70.36.8 bb4b9ad9-cf23-414f-9c82-fd3b97d6f24b False' >2012-11-08 06:06:06,729 WARN [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp-/127.0.0.1:8702-6) [2e50013e] CanDoAction of action UpdateVds failed. Reasons:VDS_CANNOT_INSTALL_STATUS_ILLEGAL,VAR__ACTION__UPDATE,VAR__TYPE__HOST >2012-11-08 06:06:06,743 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-42) Installation of 10.70.36.8. Received message: <BSTRAP component='INSTALLER' status='OK' message='Test platform succeeded'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 06:06:08,871 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-42) Installation of 10.70.36.8. Received message: <BSTRAP component='SET_SYSTEM_TIME' status='OK' message='setSystemTime ended successfully'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 06:06:11,638 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-42) Installation of 10.70.36.8. Received message: <BSTRAP component='RHN_REGISTRATION' status='OK' message='Host properly registered with RHN/Satellite.'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 06:06:11,807 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-42) Installation of 10.70.36.8. Received message: <BSTRAP component='VDSM_MAJOR_VER' status='OK' message='Available VDSM matches requirements'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 06:06:11,812 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-42) Installation of 10.70.36.8. Received message: <BSTRAP component='VT_SVM' status='OK' processor='Intel' message='Server supports virtualization'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 06:06:11,817 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-42) Installation of 10.70.36.8. Received message: <BSTRAP component='OS' status='OK' type='RHEL6' message='Supported platform version'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 06:06:11,822 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-42) Installation of 10.70.36.8. Received message: <BSTRAP component='KERNEL' status='OK' version='279' message='Supported kernel version: 279'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 06:06:11,917 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-42) Installation of 10.70.36.8. Received message: <BSTRAP component='REQ PACKAGES' status='OK' result='SDL.x86_64' message='SDL-1.2.14-3.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 06:06:11,935 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-42) Installation of 10.70.36.8. Received message: <BSTRAP component='REQ PACKAGES' status='OK' result='bridge-utils.x86_64' message='bridge-utils-1.2-9.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 06:06:11,951 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-42) Installation of 10.70.36.8. Received message: <BSTRAP component='REQ PACKAGES' status='OK' result='mesa-libGLU.x86_64' message='mesa-libGLU-7.11-5.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 06:06:11,969 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-42) Installation of 10.70.36.8. Received message: <BSTRAP component='REQ PACKAGES' status='OK' result='openssl.x86_64' message='openssl-1.0.0-25.el6_3.1.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 06:06:11,987 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-42) Installation of 10.70.36.8. Received message: <BSTRAP component='REQ PACKAGES' status='OK' result='m2crypto.x86_64' message='m2crypto-0.20.2-9.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 06:06:12,008 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-42) Installation of 10.70.36.8. Received message: <BSTRAP component='REQ PACKAGES' status='OK' result='rsync.x86_64' message='rsync-3.0.6-9.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 06:06:12,022 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-42) Installation of 10.70.36.8. Received message: <BSTRAP component='VDS PACKAGES' status='WARN' result='qemu-kvm' message='package qemu-kvm is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 06:06:12,035 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-42) Installation of 10.70.36.8. Received message: <BSTRAP component='VDS PACKAGES' status='WARN' result='qemu-kvm-tools' message='package qemu-kvm-tools is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 06:06:12,054 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-42) Installation of 10.70.36.8. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='(~vdsm~, ~vdsm-cli~)' message='vdsm-4.9.6-41.0.el6_3.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 06:06:12,068 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-42) Installation of 10.70.36.8. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='libjpeg' message='libjpeg-6b-46.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 06:06:12,083 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-42) Installation of 10.70.36.8. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='spice-server' message='spice-server-0.10.1-10.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 06:06:12,098 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-42) Installation of 10.70.36.8. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='pixman' message='pixman-0.18.4-1.el6_0.1.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 06:06:12,126 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-42) Installation of 10.70.36.8. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='seabios' message='seabios-0.6.1.2-19.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 06:06:12,134 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-42) Installation of 10.70.36.8. Received message: <BSTRAP component='VDS PACKAGES' status='WARN' result='qemu-img' message='package qemu-img is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 06:06:12,150 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-42) Installation of 10.70.36.8. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='fence-agents' message='fence-agents-3.1.5-17.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 06:06:12,174 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-42) Installation of 10.70.36.8. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='libselinux-python' message='libselinux-python-2.0.94-5.3.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 06:06:12,189 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-42) Installation of 10.70.36.8. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='sanlock' message='sanlock-2.3-4.el6_3.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 06:06:12,203 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-42) Installation of 10.70.36.8. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='sanlock-python' message='sanlock-python-2.3-4.el6_3.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 06:06:41,995 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-38) Installation of 10.70.36.15. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='(~vdsm~, ~vdsm-cli~)' message='vdsm-4.9.6-41.0.el6_3.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 06:06:44,200 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-38) Installation of 10.70.36.15. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='libjpeg' message='libjpeg-6b-46.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 06:06:46,430 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-38) Installation of 10.70.36.15. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='spice-server' message='spice-server-0.10.1-10.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 06:06:48,698 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-38) Installation of 10.70.36.15. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='pixman' message='pixman-0.18.4-1.el6_0.1.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 06:06:50,930 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-38) Installation of 10.70.36.15. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='seabios' message='seabios-0.6.1.2-19.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 06:06:53,167 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-38) Installation of 10.70.36.15. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='fence-agents' message='fence-agents-3.1.5-17.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 06:06:55,417 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-38) Installation of 10.70.36.15. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='libselinux-python' message='libselinux-python-2.0.94-5.3.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 06:06:57,629 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-38) Installation of 10.70.36.15. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='sanlock' message='sanlock-2.3-4.el6_3.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 06:06:59,907 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-38) Installation of 10.70.36.15. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='sanlock-python' message='sanlock-python-2.3-4.el6_3.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 06:07:02,602 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-38) Installation of 10.70.36.15. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='qemu-img' message='package qemu-img is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 06:07:05,091 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-38) Installation of 10.70.36.15. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='qemu-kvm-tools' message='package qemu-kvm-tools is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 06:07:07,306 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-42) Installation of 10.70.36.8. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='(~vdsm~, ~vdsm-cli~)' message='vdsm-4.9.6-41.0.el6_3.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 06:07:07,368 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-38) Installation of 10.70.36.15. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='qemu-kvm' message='package qemu-kvm is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 06:07:07,373 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-38) Installation of 10.70.36.15. Received message: <BSTRAP component='CreateConf' status='OK' message='Basic configuration found, skipping this step'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 06:07:07,378 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-38) Installation of 10.70.36.15. Received message: <BSTRAP component='CheckLocalHostname' status='OK' message='Local hostname is correct.'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 06:07:09,729 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-42) Installation of 10.70.36.8. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='libjpeg' message='libjpeg-6b-46.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 06:07:12,228 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-42) Installation of 10.70.36.8. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='spice-server' message='spice-server-0.10.1-10.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 06:07:14,437 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-38) Installation of 10.70.36.15. Received message: <BSTRAP component='SetNetworking' status='OK' message='Created management bridge.'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 06:07:14,556 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-38) Installation of 10.70.36.15. Received message: <BSTRAP component='SetSSHAccess' status='OK' message='SUCCESS'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 06:07:14,641 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-42) Installation of 10.70.36.8. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='pixman' message='pixman-0.18.4-1.el6_0.1.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 06:07:14,820 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-38) Installation of 10.70.36.15. Received message: <BSTRAP component='VerifyServices' status='OK' message='Needed services set'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 06:07:17,042 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-42) Installation of 10.70.36.8. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='seabios' message='seabios-0.6.1.2-19.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 06:07:19,573 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-42) Installation of 10.70.36.8. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='fence-agents' message='fence-agents-3.1.5-17.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 06:07:22,870 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-42) Installation of 10.70.36.8. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='libselinux-python' message='libselinux-python-2.0.94-5.3.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 06:07:24,003 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-38) Installation of 10.70.36.15. Received message: <BSTRAP component='Encryption setup' status='OK' message='Ended successfully'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 06:07:24,010 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-38) Installation of 10.70.36.15. Received message: <BSTRAP component='RHEV_INSTALL' status='OK'/>. Stage completed. (Stage: Running first installation script on Host) >2012-11-08 06:07:24,041 ERROR [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-49) [6f7c3581] Installation of 10.70.36.15. Error: Repository JBOSS is listed more than once in the configuration >Repository qa-latest-5 is listed more than once in the configuration >Repository rhel-source is listed more than once in the configuration >Repository rhel-source-beta is listed more than once in the configuration >Repository rhs-client is listed more than once in the configuration >Repository JBOSS is listed more than once in the configuration >Repository qa-latest-5 is listed more than once in the configuration >Repository rhel-source is listed more than once in the configuration >Repository rhel-source-beta is listed more than once in the configuration >Repository rhs-client is listed more than once in the configuration >. (Stage: Downloading certificate request from Host) >2012-11-08 06:07:24,042 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-49) [6f7c3581] VDS message: Repository JBOSS is listed more than once in the configuration >2012-11-08 06:07:24,042 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-49) [6f7c3581] VDS message: Repository qa-latest-5 is listed more than once in the configuration >2012-11-08 06:07:24,042 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-49) [6f7c3581] VDS message: Repository rhel-source is listed more than once in the configuration >2012-11-08 06:07:24,042 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-49) [6f7c3581] VDS message: Repository rhel-source-beta is listed more than once in the configuration >2012-11-08 06:07:24,042 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-49) [6f7c3581] VDS message: Repository rhs-client is listed more than once in the configuration >2012-11-08 06:07:24,042 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-49) [6f7c3581] VDS message: Repository JBOSS is listed more than once in the configuration >2012-11-08 06:07:24,042 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-49) [6f7c3581] VDS message: Repository qa-latest-5 is listed more than once in the configuration >2012-11-08 06:07:24,042 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-49) [6f7c3581] VDS message: Repository rhel-source is listed more than once in the configuration >2012-11-08 06:07:24,042 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-49) [6f7c3581] VDS message: Repository rhel-source-beta is listed more than once in the configuration >2012-11-08 06:07:24,042 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-49) [6f7c3581] VDS message: Repository rhs-client is listed more than once in the configuration >2012-11-08 06:07:24,042 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-49) [6f7c3581] Script ended, result is {1} >2012-11-08 06:07:24,042 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-49) [6f7c3581] Installation of 10.70.36.15. Executing installation stage. (Stage: Downloading certificate request from Host) >2012-11-08 06:07:24,042 INFO [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-4-thread-49) SSH receive 10.70.36.15:'/tmp/cert_014162ae-00de-44fa-8855-4bbafa8d6378.req' '/etc/pki/ovirt-engine/requests/cert_014162ae-00de-44fa-8855-4bbafa8d6378.req') >2012-11-08 06:07:24,131 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-49) [6f7c3581] Installation of 10.70.36.15. successfully done sftp operation ( Stage: Downloading certificate request from Host) >2012-11-08 06:07:24,131 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-49) [6f7c3581] DownloadCertificateRequest ended:true >2012-11-08 06:07:24,131 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-49) [6f7c3581] Installation of 10.70.36.15. Executing installation stage. (Stage: Sign certificate request and generate certificate) >2012-11-08 06:07:25,184 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-49) [6f7c3581] SignCertificateRequest ended:true >2012-11-08 06:07:25,184 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-49) [6f7c3581] Installation of 10.70.36.15. Executing installation stage. (Stage: Upload signed sertificate to Host) >2012-11-08 06:07:25,210 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-49) [6f7c3581] Installation of 10.70.36.15. successfully done sftp operation ( Stage: Upload signed sertificate to Host) >2012-11-08 06:07:25,211 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-49) [6f7c3581] UploadSignedCertificate ended:true >2012-11-08 06:07:25,211 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-49) [6f7c3581] Installation of 10.70.36.15. Executing installation stage. (Stage: Upload Cerficate Autority to Host) >2012-11-08 06:07:25,269 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-49) [6f7c3581] Installation of 10.70.36.15. successfully done sftp operation ( Stage: Upload Cerficate Autority to Host) >2012-11-08 06:07:25,269 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-49) [6f7c3581] Installation of 10.70.36.15. Executing installation stage. (Stage: Running second installation script on Host) >2012-11-08 06:07:25,269 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-49) [6f7c3581] Installation of 10.70.36.15. Sending SSH Command umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar -C "${MYTMP}" --no-same-permissions -o -x && "${MYTMP}"/setup -c 'ssl=true;management_port=54321' -O 'lab.eng.blr.redhat.com' -t 2012-11-08T11:05:58 -S /tmp/ovirt-id_rsa_014162ae-00de-44fa-8855-4bbafa8d6378 -p 80 -b -B rhevm http://guido.lab.eng.blr.redhat.com:80/Components/vds/ http://guido.lab.eng.blr.redhat.com:80/Components/vds/ 10.70.36.15 014162ae-00de-44fa-8855-4bbafa8d6378 True < /var/cache/ovirt-engine/vdsm-bootstrap-2.tar. (Stage: Running second installation script on Host) >2012-11-08 06:07:25,269 INFO [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-4-thread-49) SSH execute 10.70.36.15 'umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar -C "${MYTMP}" --no-same-permissions -o -x && "${MYTMP}"/setup -c 'ssl=true;management_port=54321' -O 'lab.eng.blr.redhat.com' -t 2012-11-08T11:05:58 -S /tmp/ovirt-id_rsa_014162ae-00de-44fa-8855-4bbafa8d6378 -p 80 -b -B rhevm http://guido.lab.eng.blr.redhat.com:80/Components/vds/ http://guido.lab.eng.blr.redhat.com:80/Components/vds/ 10.70.36.15 014162ae-00de-44fa-8855-4bbafa8d6378 True' >2012-11-08 06:07:25,360 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-38) Installation of 10.70.36.15. Received message: <BSTRAP component='INSTALLER' status='OK' message='Test platform succeeded'/>. FYI. (Stage: Running second installation script on Host) >2012-11-08 06:07:25,698 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-42) Installation of 10.70.36.8. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='sanlock' message='sanlock-2.3-4.el6_3.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 06:07:25,700 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-38) Installation of 10.70.36.15. Received message: <BSTRAP component='instCert' status='OK'/>. FYI. (Stage: Running second installation script on Host) >2012-11-08 06:07:25,706 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-38) Installation of 10.70.36.15. Received message: <BSTRAP component='CoreDump' status='OK'/>. FYI. (Stage: Running second installation script on Host) >2012-11-08 06:07:25,711 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-38) Installation of 10.70.36.15. Received message: <BSTRAP component='cleanAll' status='OK'/>. FYI. (Stage: Running second installation script on Host) >2012-11-08 06:07:25,732 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-38) Installation of 10.70.36.15. Received message: <BSTRAP component='VDS Configuration' status='OK'/>. FYI. (Stage: Running second installation script on Host) >2012-11-08 06:07:25,736 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-38) Installation of 10.70.36.15. Received message: <BSTRAP component='RHEV_INSTALL' status='OK'/>. Stage completed. (Stage: Running second installation script on Host) >2012-11-08 06:07:25,741 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-38) Installation of 10.70.36.15. Received message: <BSTRAP component='Reboot' status='OK' message='Rebooting machine' />. FYI. (Stage: Host installation complete) >2012-11-08 06:07:25,747 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-49) [6f7c3581] Script ended, result is {1} >2012-11-08 06:07:25,748 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-4-thread-49) [6f7c3581] After Installation pool-4-thread-49 >2012-11-08 06:07:25,749 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-49) [6f7c3581] START, SetVdsStatusVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, status=Reboot, nonOperationalReason=NONE), log id: 4c0c2129 >2012-11-08 06:07:25,755 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-49) [6f7c3581] FINISH, SetVdsStatusVDSCommand, log id: 4c0c2129 >2012-11-08 06:07:25,756 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-4-thread-48) [6f7c3581] Waiting 300 seconds, for server to finish reboot process. >2012-11-08 06:07:28,622 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-42) Installation of 10.70.36.8. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='sanlock-python' message='sanlock-python-2.3-4.el6_3.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 06:07:31,196 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-42) Installation of 10.70.36.8. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='qemu-img' message='package qemu-img is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 06:07:33,764 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-42) Installation of 10.70.36.8. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='qemu-kvm-tools' message='package qemu-kvm-tools is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 06:07:36,236 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-42) Installation of 10.70.36.8. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='qemu-kvm' message='package qemu-kvm is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 06:07:36,242 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-42) Installation of 10.70.36.8. Received message: <BSTRAP component='CreateConf' status='OK' message='Basic configuration found, skipping this step'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 06:07:36,247 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-42) Installation of 10.70.36.8. Received message: <BSTRAP component='CheckLocalHostname' status='OK' message='Local hostname is correct.'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 06:07:47,415 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-42) Installation of 10.70.36.8. Received message: <BSTRAP component='SetNetworking' status='OK' message='Created management bridge.'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 06:07:47,524 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-42) Installation of 10.70.36.8. Received message: <BSTRAP component='SetSSHAccess' status='OK' message='SUCCESS'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 06:07:47,781 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-42) Installation of 10.70.36.8. Received message: <BSTRAP component='VerifyServices' status='OK' message='Needed services set'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 06:07:51,238 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-42) Installation of 10.70.36.8. Received message: <BSTRAP component='Encryption setup' status='OK' message='Ended successfully'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 06:07:51,243 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-42) Installation of 10.70.36.8. Received message: <BSTRAP component='RHEV_INSTALL' status='OK'/>. Stage completed. (Stage: Running first installation script on Host) >2012-11-08 06:07:51,275 ERROR [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-50) [26b62dee] Installation of 10.70.36.8. Error: Repository JBOSS is listed more than once in the configuration >Repository qa-latest-5 is listed more than once in the configuration >Repository rhel-source is listed more than once in the configuration >Repository rhel-source-beta is listed more than once in the configuration >Repository rhs-client is listed more than once in the configuration >Repository JBOSS is listed more than once in the configuration >Repository qa-latest-5 is listed more than once in the configuration >Repository rhel-source is listed more than once in the configuration >Repository rhel-source-beta is listed more than once in the configuration >Repository rhs-client is listed more than once in the configuration >. (Stage: Downloading certificate request from Host) >2012-11-08 06:07:51,275 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-50) [26b62dee] VDS message: Repository JBOSS is listed more than once in the configuration >2012-11-08 06:07:51,275 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-50) [26b62dee] VDS message: Repository qa-latest-5 is listed more than once in the configuration >2012-11-08 06:07:51,275 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-50) [26b62dee] VDS message: Repository rhel-source is listed more than once in the configuration >2012-11-08 06:07:51,275 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-50) [26b62dee] VDS message: Repository rhel-source-beta is listed more than once in the configuration >2012-11-08 06:07:51,275 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-50) [26b62dee] VDS message: Repository rhs-client is listed more than once in the configuration >2012-11-08 06:07:51,275 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-50) [26b62dee] VDS message: Repository JBOSS is listed more than once in the configuration >2012-11-08 06:07:51,275 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-50) [26b62dee] VDS message: Repository qa-latest-5 is listed more than once in the configuration >2012-11-08 06:07:51,276 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-50) [26b62dee] VDS message: Repository rhel-source is listed more than once in the configuration >2012-11-08 06:07:51,276 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-50) [26b62dee] VDS message: Repository rhel-source-beta is listed more than once in the configuration >2012-11-08 06:07:51,276 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-50) [26b62dee] VDS message: Repository rhs-client is listed more than once in the configuration >2012-11-08 06:07:51,276 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-50) [26b62dee] Script ended, result is {1} >2012-11-08 06:07:51,276 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-50) [26b62dee] Installation of 10.70.36.8. Executing installation stage. (Stage: Downloading certificate request from Host) >2012-11-08 06:07:51,276 INFO [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-4-thread-50) SSH receive 10.70.36.8:'/tmp/cert_bb4b9ad9-cf23-414f-9c82-fd3b97d6f24b.req' '/etc/pki/ovirt-engine/requests/cert_bb4b9ad9-cf23-414f-9c82-fd3b97d6f24b.req') >2012-11-08 06:07:51,338 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-50) [26b62dee] Installation of 10.70.36.8. successfully done sftp operation ( Stage: Downloading certificate request from Host) >2012-11-08 06:07:51,338 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-50) [26b62dee] DownloadCertificateRequest ended:true >2012-11-08 06:07:51,338 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-50) [26b62dee] Installation of 10.70.36.8. Executing installation stage. (Stage: Sign certificate request and generate certificate) >2012-11-08 06:07:52,341 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-50) [26b62dee] SignCertificateRequest ended:true >2012-11-08 06:07:52,341 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-50) [26b62dee] Installation of 10.70.36.8. Executing installation stage. (Stage: Upload signed sertificate to Host) >2012-11-08 06:07:52,368 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-50) [26b62dee] Installation of 10.70.36.8. successfully done sftp operation ( Stage: Upload signed sertificate to Host) >2012-11-08 06:07:52,368 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-50) [26b62dee] UploadSignedCertificate ended:true >2012-11-08 06:07:52,368 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-50) [26b62dee] Installation of 10.70.36.8. Executing installation stage. (Stage: Upload Cerficate Autority to Host) >2012-11-08 06:07:52,425 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-50) [26b62dee] Installation of 10.70.36.8. successfully done sftp operation ( Stage: Upload Cerficate Autority to Host) >2012-11-08 06:07:52,425 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-50) [26b62dee] Installation of 10.70.36.8. Executing installation stage. (Stage: Running second installation script on Host) >2012-11-08 06:07:52,425 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-50) [26b62dee] Installation of 10.70.36.8. Sending SSH Command umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar -C "${MYTMP}" --no-same-permissions -o -x && "${MYTMP}"/setup -c 'ssl=true;management_port=54321' -O 'lab.eng.blr.redhat.com' -t 2012-11-08T11:06:06 -S /tmp/ovirt-id_rsa_bb4b9ad9-cf23-414f-9c82-fd3b97d6f24b -p 80 -b -B rhevm http://guido.lab.eng.blr.redhat.com:80/Components/vds/ http://guido.lab.eng.blr.redhat.com:80/Components/vds/ 10.70.36.8 bb4b9ad9-cf23-414f-9c82-fd3b97d6f24b True < /var/cache/ovirt-engine/vdsm-bootstrap-2.tar. (Stage: Running second installation script on Host) >2012-11-08 06:07:52,425 INFO [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-4-thread-50) SSH execute 10.70.36.8 'umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar -C "${MYTMP}" --no-same-permissions -o -x && "${MYTMP}"/setup -c 'ssl=true;management_port=54321' -O 'lab.eng.blr.redhat.com' -t 2012-11-08T11:06:06 -S /tmp/ovirt-id_rsa_bb4b9ad9-cf23-414f-9c82-fd3b97d6f24b -p 80 -b -B rhevm http://guido.lab.eng.blr.redhat.com:80/Components/vds/ http://guido.lab.eng.blr.redhat.com:80/Components/vds/ 10.70.36.8 bb4b9ad9-cf23-414f-9c82-fd3b97d6f24b True' >2012-11-08 06:07:52,518 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-42) Installation of 10.70.36.8. Received message: <BSTRAP component='INSTALLER' status='OK' message='Test platform succeeded'/>. FYI. (Stage: Running second installation script on Host) >2012-11-08 06:07:52,862 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-42) Installation of 10.70.36.8. Received message: <BSTRAP component='instCert' status='OK'/>. FYI. (Stage: Running second installation script on Host) >2012-11-08 06:07:52,867 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-42) Installation of 10.70.36.8. Received message: <BSTRAP component='CoreDump' status='OK'/>. FYI. (Stage: Running second installation script on Host) >2012-11-08 06:07:52,871 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-42) Installation of 10.70.36.8. Received message: <BSTRAP component='cleanAll' status='OK'/>. FYI. (Stage: Running second installation script on Host) >2012-11-08 06:07:52,876 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-42) Installation of 10.70.36.8. Received message: <BSTRAP component='VDS Configuration' status='OK'/>. FYI. (Stage: Running second installation script on Host) >2012-11-08 06:07:52,878 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-42) Installation of 10.70.36.8. Received message: <BSTRAP component='RHEV_INSTALL' status='OK'/>. Stage completed. (Stage: Running second installation script on Host) >2012-11-08 06:07:52,882 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-42) Installation of 10.70.36.8. Received message: <BSTRAP component='Reboot' status='OK' message='Rebooting machine' />. FYI. (Stage: Host installation complete) >2012-11-08 06:07:52,898 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-50) [26b62dee] Script ended, result is {1} >2012-11-08 06:07:52,898 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-4-thread-50) [26b62dee] After Installation pool-4-thread-50 >2012-11-08 06:07:52,911 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-50) [26b62dee] START, SetVdsStatusVDSCommand(HostName = rhs-gp-srv4.lab.eng.blr.redhat.com, HostId = d2bd32d4-2992-11e2-ab59-52540035ed48, status=Reboot, nonOperationalReason=NONE), log id: 3f1137e >2012-11-08 06:07:52,917 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-50) [26b62dee] FINISH, SetVdsStatusVDSCommand, log id: 3f1137e >2012-11-08 06:07:52,917 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-4-thread-49) [26b62dee] Waiting 300 seconds, for server to finish reboot process. >2012-11-08 06:10:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-17) Autorecovering hosts is disabled, skipping >2012-11-08 06:10:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-17) Autorecovering storage domains is disabled, skipping >2012-11-08 06:10:06,273 INFO [org.ovirt.engine.core.bll.storage.FenceVdsManualyCommand] (ajp-/127.0.0.1:8702-11) Lock Acquired to object EngineLock [exclusiveLocks= key: 96739be4-2990-11e2-8577-52540035ed48 value: VDS_FENCE >, sharedLocks= ] >2012-11-08 06:10:06,273 INFO [org.ovirt.engine.core.bll.storage.FenceVdsManualyCommand] (pool-4-thread-50) [736dc77d] Running command: FenceVdsManualyCommand internal: false. Entities affected : ID: 96739be4-2990-11e2-8577-52540035ed48 Type: VDS >2012-11-08 06:10:06,278 INFO [org.ovirt.engine.core.bll.ClearNonResponsiveVdsVmsCommand] (pool-4-thread-50) [421aec98] Running command: ClearNonResponsiveVdsVmsCommand internal: true. Entities affected : ID: 96739be4-2990-11e2-8577-52540035ed48 Type: VDS >2012-11-08 06:10:06,281 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsVMsClearedVDSCommand] (pool-4-thread-50) [421aec98] START, UpdateVdsVMsClearedVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48), log id: 52a4774d >2012-11-08 06:10:06,283 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsVMsClearedVDSCommand] (pool-4-thread-50) [421aec98] FINISH, UpdateVdsVMsClearedVDSCommand, log id: 52a4774d >2012-11-08 06:10:06,289 INFO [org.ovirt.engine.core.bll.storage.FenceVdsManualyCommand] (pool-4-thread-50) [421aec98] Lock freed to object EngineLock [exclusiveLocks= key: 96739be4-2990-11e2-8577-52540035ed48 value: VDS_FENCE >, sharedLocks= ] >2012-11-08 06:10:16,573 INFO [org.ovirt.engine.core.bll.storage.FenceVdsManualyCommand] (ajp-/127.0.0.1:8702-13) Lock Acquired to object EngineLock [exclusiveLocks= key: d2bd32d4-2992-11e2-ab59-52540035ed48 value: VDS_FENCE >, sharedLocks= ] >2012-11-08 06:10:16,574 INFO [org.ovirt.engine.core.bll.storage.FenceVdsManualyCommand] (pool-4-thread-50) [b2b0e61] Running command: FenceVdsManualyCommand internal: false. Entities affected : ID: d2bd32d4-2992-11e2-ab59-52540035ed48 Type: VDS >2012-11-08 06:10:16,577 INFO [org.ovirt.engine.core.bll.ClearNonResponsiveVdsVmsCommand] (pool-4-thread-50) [23c9df38] Running command: ClearNonResponsiveVdsVmsCommand internal: true. Entities affected : ID: d2bd32d4-2992-11e2-ab59-52540035ed48 Type: VDS >2012-11-08 06:10:16,579 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsVMsClearedVDSCommand] (pool-4-thread-50) [23c9df38] START, UpdateVdsVMsClearedVDSCommand(HostName = rhs-gp-srv4.lab.eng.blr.redhat.com, HostId = d2bd32d4-2992-11e2-ab59-52540035ed48), log id: 292d3556 >2012-11-08 06:10:16,580 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsVMsClearedVDSCommand] (pool-4-thread-50) [23c9df38] FINISH, UpdateVdsVMsClearedVDSCommand, log id: 292d3556 >2012-11-08 06:10:16,586 INFO [org.ovirt.engine.core.bll.storage.FenceVdsManualyCommand] (pool-4-thread-50) [23c9df38] Lock freed to object EngineLock [exclusiveLocks= key: d2bd32d4-2992-11e2-ab59-52540035ed48 value: VDS_FENCE >, sharedLocks= ] >2012-11-08 06:12:25,758 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-48) [6f7c3581] START, SetVdsStatusVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, status=NonResponsive, nonOperationalReason=NONE), log id: 23234547 >2012-11-08 06:12:25,762 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-48) [6f7c3581] FINISH, SetVdsStatusVDSCommand, log id: 23234547 >2012-11-08 06:12:27,726 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-71) [206fd542] Running command: InitVdsOnUpCommand internal: true. >2012-11-08 06:12:27,733 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-71) [206fd542] No string for UNASSIGNED type. Use default Log >2012-11-08 06:12:27,742 INFO [org.ovirt.engine.core.bll.MultipleActionsRunner] (pool-4-thread-48) [206fd542] MultipleActionsRunner of type MigrateVmToServer invoked with no actions >2012-11-08 06:12:27,752 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-71) [7886405d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 96739be4-2990-11e2-8577-52540035ed48 Type: VDS >2012-11-08 06:12:27,756 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-71) [3e904d2f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 96739be4-2990-11e2-8577-52540035ed48 Type: VDS >2012-11-08 06:12:52,920 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-49) [26b62dee] START, SetVdsStatusVDSCommand(HostName = rhs-gp-srv4.lab.eng.blr.redhat.com, HostId = d2bd32d4-2992-11e2-ab59-52540035ed48, status=NonResponsive, nonOperationalReason=NONE), log id: 1845cf90 >2012-11-08 06:12:52,925 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-49) [26b62dee] FINISH, SetVdsStatusVDSCommand, log id: 1845cf90 >2012-11-08 06:12:54,484 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-17) [7dc3d7df] Running command: InitVdsOnUpCommand internal: true. >2012-11-08 06:12:54,490 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-17) [7dc3d7df] No string for UNASSIGNED type. Use default Log >2012-11-08 06:12:54,492 INFO [org.ovirt.engine.core.bll.MultipleActionsRunner] (pool-4-thread-49) [7dc3d7df] MultipleActionsRunner of type MigrateVmToServer invoked with no actions >2012-11-08 06:12:54,500 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-17) [f5b6db1] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: d2bd32d4-2992-11e2-ab59-52540035ed48 Type: VDS >2012-11-08 06:12:54,504 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-17) [36467d8f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: d2bd32d4-2992-11e2-ab59-52540035ed48 Type: VDS >2012-11-08 06:15:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-37) Autorecovering hosts is disabled, skipping >2012-11-08 06:15:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-37) Autorecovering storage domains is disabled, skipping >2012-11-08 06:19:55,936 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-50) DbUserCacheManager::refreshAllUserData() - entered >2012-11-08 06:19:58,470 INFO [org.ovirt.engine.core.bll.LogoutUserCommand] (ajp-/127.0.0.1:8702-9) [5e7c41da] Running command: LogoutUserCommand internal: false. >2012-11-08 06:20:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-64) Autorecovering hosts is disabled, skipping >2012-11-08 06:20:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-64) Autorecovering storage domains is disabled, skipping >2012-11-08 06:21:36,956 INFO [org.ovirt.engine.core.bll.FencingExecutor] (ajp-/127.0.0.1:8702-6) Executing <Status> Power Management command, Proxy Host:rhs-gp-srv4.lab.eng.blr.redhat.com, Agent:ipmilan, Target Host:, Management IP:10.70.33.8, User:ADMIN, Options: >2012-11-08 06:21:36,958 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp-/127.0.0.1:8702-6) START, FenceVdsVDSCommand(HostName = rhs-gp-srv4.lab.eng.blr.redhat.com, HostId = d2bd32d4-2992-11e2-ab59-52540035ed48, targetVdsId = 96739be4-2990-11e2-8577-52540035ed48, action = Status, ip = 10.70.33.8, port = , type = ipmilan, user = ADMIN, password = ******, options = ''), log id: d55b5f7 >2012-11-08 06:21:37,091 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp-/127.0.0.1:8702-6) FINISH, FenceVdsVDSCommand, return: Test Succeeded, Host Status is: on, log id: d55b5f7 >2012-11-08 06:21:40,693 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp-/127.0.0.1:8702-13) [71672a5e] Running command: UpdateVdsCommand internal: false. Entities affected : ID: 96739be4-2990-11e2-8577-52540035ed48 Type: VDS >2012-11-08 06:21:40,812 INFO [org.ovirt.engine.core.bll.FencingExecutor] (ajp-/127.0.0.1:8702-13) [71672a5e] Executing <Status> Power Management command, Proxy Host:rhs-gp-srv4.lab.eng.blr.redhat.com, Agent:ipmilan, Target Host:rhs-gp-srv11.lab.eng.blr.redhat.com, Management IP:10.70.33.8, User:ADMIN, Options: >2012-11-08 06:21:40,817 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp-/127.0.0.1:8702-13) [71672a5e] START, FenceVdsVDSCommand(HostName = rhs-gp-srv4.lab.eng.blr.redhat.com, HostId = d2bd32d4-2992-11e2-ab59-52540035ed48, targetVdsId = 96739be4-2990-11e2-8577-52540035ed48, action = Status, ip = 10.70.33.8, port = , type = ipmilan, user = ADMIN, password = ******, options = ''), log id: 6e1decd2 >2012-11-08 06:21:40,899 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp-/127.0.0.1:8702-13) [71672a5e] FINISH, FenceVdsVDSCommand, return: Test Succeeded, Host Status is: on, log id: 6e1decd2 >2012-11-08 06:22:09,047 INFO [org.ovirt.engine.core.bll.FencingExecutor] (ajp-/127.0.0.1:8702-4) Executing <Status> Power Management command, Proxy Host:rhs-gp-srv11.lab.eng.blr.redhat.com, Agent:ipmilan, Target Host:, Management IP:10.70.33.15, User:ADMIN, Options: >2012-11-08 06:22:09,059 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp-/127.0.0.1:8702-4) START, FenceVdsVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, targetVdsId = d2bd32d4-2992-11e2-ab59-52540035ed48, action = Status, ip = 10.70.33.15, port = , type = ipmilan, user = ADMIN, password = ******, options = ''), log id: 3300f9ee >2012-11-08 06:22:09,148 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp-/127.0.0.1:8702-4) FINISH, FenceVdsVDSCommand, return: Test Failed, Host Status is: unknown. The fence-agent script reported the following error: Getting status of IPMI:10.70.33.15...Chassis power = Unknown >Failed >, log id: 3300f9ee >2012-11-08 06:22:40,266 INFO [org.ovirt.engine.core.bll.FencingExecutor] (ajp-/127.0.0.1:8702-2) Executing <Status> Power Management command, Proxy Host:rhs-gp-srv11.lab.eng.blr.redhat.com, Agent:ipmilan, Target Host:, Management IP:10.70.33.15, User:ADMIN, Options: >2012-11-08 06:22:40,277 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp-/127.0.0.1:8702-2) START, FenceVdsVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, targetVdsId = d2bd32d4-2992-11e2-ab59-52540035ed48, action = Status, ip = 10.70.33.15, port = , type = ipmilan, user = ADMIN, password = ******, options = ''), log id: 77721513 >2012-11-08 06:22:40,356 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp-/127.0.0.1:8702-2) FINISH, FenceVdsVDSCommand, return: Test Succeeded, Host Status is: on, log id: 77721513 >2012-11-08 06:22:42,265 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp-/127.0.0.1:8702-6) [2e89731c] Running command: UpdateVdsCommand internal: false. Entities affected : ID: d2bd32d4-2992-11e2-ab59-52540035ed48 Type: VDS >2012-11-08 06:22:42,403 INFO [org.ovirt.engine.core.bll.FencingExecutor] (ajp-/127.0.0.1:8702-6) [2e89731c] Executing <Status> Power Management command, Proxy Host:rhs-gp-srv11.lab.eng.blr.redhat.com, Agent:ipmilan, Target Host:rhs-gp-srv4.lab.eng.blr.redhat.com, Management IP:10.70.33.15, User:ADMIN, Options: >2012-11-08 06:22:42,416 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp-/127.0.0.1:8702-6) [2e89731c] START, FenceVdsVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, targetVdsId = d2bd32d4-2992-11e2-ab59-52540035ed48, action = Status, ip = 10.70.33.15, port = , type = ipmilan, user = ADMIN, password = ******, options = ''), log id: 4e04c357 >2012-11-08 06:22:42,511 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (ajp-/127.0.0.1:8702-6) [2e89731c] FINISH, FenceVdsVDSCommand, return: Test Succeeded, Host Status is: on, log id: 4e04c357 >2012-11-08 06:25:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-84) Autorecovering hosts is disabled, skipping >2012-11-08 06:25:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-84) Autorecovering storage domains is disabled, skipping >2012-11-08 06:27:38,992 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (ajp-/127.0.0.1:8702-7) [1d1655e6] START, ValidateStorageServerConnectionVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = POSIXFS, connectionList = [{ id: null, connection: rhs-client36.lab.eng.blr.redhat.com:distribute, iqn: null, vfsType: glusterfs, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 2aa4047b >2012-11-08 06:27:39,015 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (ajp-/127.0.0.1:8702-7) [1d1655e6] FINISH, ValidateStorageServerConnectionVDSCommand, return: {00000000-0000-0000-0000-000000000000=0}, log id: 2aa4047b >2012-11-08 06:27:39,019 INFO [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-7) [1d1655e6] Running command: AddStorageServerConnectionCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2012-11-08 06:27:39,038 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-7) [1d1655e6] START, ConnectStorageServerVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = POSIXFS, connectionList = [{ id: null, connection: rhs-client36.lab.eng.blr.redhat.com:distribute, iqn: null, vfsType: glusterfs, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 8edf932 >2012-11-08 06:27:43,615 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-7) [1d1655e6] FINISH, ConnectStorageServerVDSCommand, return: {00000000-0000-0000-0000-000000000000=0}, log id: 8edf932 >2012-11-08 06:27:44,177 WARN [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (ajp-/127.0.0.1:8702-8) [886c80e] The message key AddPosixFsStorageDomain is missing from bundles/ExecutionMessages >2012-11-08 06:27:44,211 INFO [org.ovirt.engine.core.bll.storage.AddPosixFsStorageDomainCommand] (ajp-/127.0.0.1:8702-8) [886c80e] Running command: AddPosixFsStorageDomainCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2012-11-08 06:27:44,239 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-8) [886c80e] START, CreateStorageDomainVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storageDomain=org.ovirt.engine.core.common.businessentities.storage_domain_static@eea4df1b, args=rhs-client36.lab.eng.blr.redhat.com:distribute), log id: 3c009b34 >2012-11-08 06:27:47,113 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-8) [886c80e] FINISH, CreateStorageDomainVDSCommand, log id: 3c009b34 >2012-11-08 06:27:47,126 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStorageDomainStatsVDSCommand] (ajp-/127.0.0.1:8702-8) [886c80e] START, GetStorageDomainStatsVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storageDomainId=a02e96be-cf4c-4d0f-8368-fd9d4aa659bf), log id: 52d00b6e >2012-11-08 06:27:47,158 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStorageDomainStatsVDSCommand] (ajp-/127.0.0.1:8702-8) [886c80e] FINISH, GetStorageDomainStatsVDSCommand, return: org.ovirt.engine.core.common.businessentities.storage_domains@d71128c1, log id: 52d00b6e >2012-11-08 06:27:47,762 INFO [org.ovirt.engine.core.bll.storage.AttachStorageDomainToPoolCommand] (ajp-/127.0.0.1:8702-7) [213c6912] Running command: AttachStorageDomainToPoolCommand internal: false. Entities affected : ID: a02e96be-cf4c-4d0f-8368-fd9d4aa659bf Type: Storage >2012-11-08 06:27:47,778 INFO [org.ovirt.engine.core.bll.storage.AddStoragePoolWithStoragesCommand] (ajp-/127.0.0.1:8702-7) [61ca98d6] Lock Acquired to object EngineLock [exclusiveLocks= key: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 value: POOL >, sharedLocks= ] >2012-11-08 06:27:47,813 INFO [org.ovirt.engine.core.bll.storage.AddStoragePoolWithStoragesCommand] (ajp-/127.0.0.1:8702-7) [61ca98d6] Running command: AddStoragePoolWithStoragesCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 06:27:47,921 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (ajp-/127.0.0.1:8702-7) [6e6b1d0c] START, ValidateStorageServerConnectionVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = POSIXFS, connectionList = [{ id: 301f13d5-f9ce-47d1-a2ef-ec72ec607bac, connection: rhs-client36.lab.eng.blr.redhat.com:distribute, iqn: null, vfsType: glusterfs, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 2789dbc6 >2012-11-08 06:27:47,940 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (ajp-/127.0.0.1:8702-7) [6e6b1d0c] FINISH, ValidateStorageServerConnectionVDSCommand, return: {301f13d5-f9ce-47d1-a2ef-ec72ec607bac=0}, log id: 2789dbc6 >2012-11-08 06:27:47,940 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (ajp-/127.0.0.1:8702-7) [6e6b1d0c] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2012-11-08 06:27:47,950 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-7) [6e6b1d0c] START, ConnectStorageServerVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = POSIXFS, connectionList = [{ id: 301f13d5-f9ce-47d1-a2ef-ec72ec607bac, connection: rhs-client36.lab.eng.blr.redhat.com:distribute, iqn: null, vfsType: glusterfs, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 44f370df >2012-11-08 06:27:47,970 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-7) [6e6b1d0c] FINISH, ConnectStorageServerVDSCommand, return: {301f13d5-f9ce-47d1-a2ef-ec72ec607bac=0}, log id: 44f370df >2012-11-08 06:27:47,981 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStoragePoolVDSCommand] (ajp-/127.0.0.1:8702-7) [6e6b1d0c] START, CreateStoragePoolVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId=3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, storageType=POSIXFS, storagePoolName=RHS-REBALANCE, masterDomainId=a02e96be-cf4c-4d0f-8368-fd9d4aa659bf, domainsIdList=[a02e96be-cf4c-4d0f-8368-fd9d4aa659bf], masterVersion=1), log id: 981b81d >2012-11-08 06:28:11,301 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStoragePoolVDSCommand] (ajp-/127.0.0.1:8702-7) [6e6b1d0c] FINISH, CreateStoragePoolVDSCommand, log id: 981b81d >2012-11-08 06:28:11,304 INFO [org.ovirt.engine.core.bll.storage.AddStoragePoolWithStoragesCommand] (ajp-/127.0.0.1:8702-7) [6e6b1d0c] Lock freed to object EngineLock [exclusiveLocks= key: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 value: POOL >, sharedLocks= ] >2012-11-08 06:28:11,307 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp-/127.0.0.1:8702-7) [3573169] Lock Acquired to object EngineLock [exclusiveLocks= key: a02e96be-cf4c-4d0f-8368-fd9d4aa659bf value: STORAGE >, sharedLocks= ] >2012-11-08 06:28:11,313 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp-/127.0.0.1:8702-7) [3573169] Running command: ActivateStorageDomainCommand internal: true. Entities affected : ID: a02e96be-cf4c-4d0f-8368-fd9d4aa659bf Type: Storage >2012-11-08 06:28:11,329 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp-/127.0.0.1:8702-7) [3573169] Lock freed to object EngineLock [exclusiveLocks= key: a02e96be-cf4c-4d0f-8368-fd9d4aa659bf value: STORAGE >, sharedLocks= ] >2012-11-08 06:28:11,329 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp-/127.0.0.1:8702-7) [3573169] ActivateStorage Domain. Before Connect all hosts to pool. Time:11/8/12 6:28 AM >2012-11-08 06:28:11,405 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (pool-4-thread-49) [1720cdd3] START, ValidateStorageServerConnectionVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = POSIXFS, connectionList = [{ id: 301f13d5-f9ce-47d1-a2ef-ec72ec607bac, connection: rhs-client36.lab.eng.blr.redhat.com:distribute, iqn: null, vfsType: glusterfs, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 39b284be >2012-11-08 06:28:11,406 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (pool-4-thread-50) [75f060db] START, ValidateStorageServerConnectionVDSCommand(HostName = rhs-gp-srv4.lab.eng.blr.redhat.com, HostId = d2bd32d4-2992-11e2-ab59-52540035ed48, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = POSIXFS, connectionList = [{ id: 301f13d5-f9ce-47d1-a2ef-ec72ec607bac, connection: rhs-client36.lab.eng.blr.redhat.com:distribute, iqn: null, vfsType: glusterfs, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 28ecdbc0 >2012-11-08 06:28:11,425 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (pool-4-thread-50) [75f060db] FINISH, ValidateStorageServerConnectionVDSCommand, return: {301f13d5-f9ce-47d1-a2ef-ec72ec607bac=0}, log id: 28ecdbc0 >2012-11-08 06:28:11,425 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (pool-4-thread-50) [75f060db] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2012-11-08 06:28:11,427 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (pool-4-thread-49) [1720cdd3] FINISH, ValidateStorageServerConnectionVDSCommand, return: {301f13d5-f9ce-47d1-a2ef-ec72ec607bac=0}, log id: 39b284be >2012-11-08 06:28:11,427 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (pool-4-thread-49) [1720cdd3] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2012-11-08 06:28:11,447 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-4-thread-49) [1720cdd3] START, ConnectStorageServerVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = POSIXFS, connectionList = [{ id: 301f13d5-f9ce-47d1-a2ef-ec72ec607bac, connection: rhs-client36.lab.eng.blr.redhat.com:distribute, iqn: null, vfsType: glusterfs, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 6bad07bc >2012-11-08 06:28:11,447 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-4-thread-50) [75f060db] START, ConnectStorageServerVDSCommand(HostName = rhs-gp-srv4.lab.eng.blr.redhat.com, HostId = d2bd32d4-2992-11e2-ab59-52540035ed48, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = POSIXFS, connectionList = [{ id: 301f13d5-f9ce-47d1-a2ef-ec72ec607bac, connection: rhs-client36.lab.eng.blr.redhat.com:distribute, iqn: null, vfsType: glusterfs, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 56f6b7d1 >2012-11-08 06:28:11,485 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-4-thread-49) [1720cdd3] FINISH, ConnectStorageServerVDSCommand, return: {301f13d5-f9ce-47d1-a2ef-ec72ec607bac=0}, log id: 6bad07bc >2012-11-08 06:28:15,719 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-4-thread-50) [75f060db] FINISH, ConnectStorageServerVDSCommand, return: {301f13d5-f9ce-47d1-a2ef-ec72ec607bac=0}, log id: 56f6b7d1 >2012-11-08 06:28:15,724 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-7) [3573169] START, ActivateStorageDomainVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null, storageDomainId = a02e96be-cf4c-4d0f-8368-fd9d4aa659bf), log id: 356588b4 >2012-11-08 06:28:15,765 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (ajp-/127.0.0.1:8702-7) [3285a8cc] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 06:28:15,838 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (ajp-/127.0.0.1:8702-7) [3285a8cc] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Unknown_Pool, storage pool RHS-REBALANCE >2012-11-08 06:28:15,861 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (ajp-/127.0.0.1:8702-7) [3285a8cc] START, ConnectStoragePoolVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, vds_spm_id = 1, masterDomainId = a02e96be-cf4c-4d0f-8368-fd9d4aa659bf, masterVersion = 1), log id: 60a21c34 >2012-11-08 06:28:18,546 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (ajp-/127.0.0.1:8702-7) [3285a8cc] FINISH, ConnectStoragePoolVDSCommand, log id: 60a21c34 >2012-11-08 06:28:18,568 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (ajp-/127.0.0.1:8702-7) [3285a8cc] hostFromVds::Connected host to pool - selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 06:28:18,577 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (ajp-/127.0.0.1:8702-7) [3285a8cc] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId -1, LVER -1 >2012-11-08 06:28:18,589 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (ajp-/127.0.0.1:8702-7) [3285a8cc] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3a49e74f >2012-11-08 06:28:18,620 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (ajp-/127.0.0.1:8702-7) [3285a8cc] spmStart polling started: taskId = d54fbc0d-ae06-43bc-8b51-25f17b74f62b >2012-11-08 06:28:40,369 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (ajp-/127.0.0.1:8702-7) [3285a8cc] spmStart polling ended: taskId = d54fbc0d-ae06-43bc-8b51-25f17b74f62b task status = finished >2012-11-08 06:28:40,391 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (ajp-/127.0.0.1:8702-7) [3285a8cc] spmStart polling ended, spm status: SPM >2012-11-08 06:28:40,406 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (ajp-/127.0.0.1:8702-7) [3285a8cc] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=d54fbc0d-ae06-43bc-8b51-25f17b74f62b), log id: 19f1400c >2012-11-08 06:28:40,424 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (ajp-/127.0.0.1:8702-7) [3285a8cc] FINISH, HSMClearTaskVDSCommand, log id: 19f1400c >2012-11-08 06:28:40,424 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (ajp-/127.0.0.1:8702-7) [3285a8cc] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@70b58426, log id: 3a49e74f >2012-11-08 06:28:40,426 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (ajp-/127.0.0.1:8702-7) [3285a8cc] Initialize Irs proxy from vds: 10.70.36.15 >2012-11-08 06:28:40,482 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-7) [3285a8cc] FINISH, ActivateStorageDomainVDSCommand, log id: 356588b4 >2012-11-08 06:28:40,483 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp-/127.0.0.1:8702-7) [3285a8cc] ActivateStorage Domain. After Connect all hosts to pool. Time:11/8/12 6:28 AM >2012-11-08 06:28:40,526 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) [3285a8cc] START, ConnectStoragePoolVDSCommand(HostName = rhs-gp-srv4.lab.eng.blr.redhat.com, HostId = d2bd32d4-2992-11e2-ab59-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, vds_spm_id = 2, masterDomainId = a02e96be-cf4c-4d0f-8368-fd9d4aa659bf, masterVersion = 1), log id: 79335e9 >2012-11-08 06:28:40,528 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-50) [3285a8cc] START, ConnectStoragePoolVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, vds_spm_id = 1, masterDomainId = a02e96be-cf4c-4d0f-8368-fd9d4aa659bf, masterVersion = 1), log id: 434985a1 >2012-11-08 06:28:40,551 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-50) [3285a8cc] FINISH, ConnectStoragePoolVDSCommand, log id: 434985a1 >2012-11-08 06:28:43,161 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) [3285a8cc] FINISH, ConnectStoragePoolVDSCommand, log id: 79335e9 >2012-11-08 06:28:43,180 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp-/127.0.0.1:8702-7) [3285a8cc] ActivateStorage Domain. After change storage pool status in vds. Time:11/8/12 6:28 AM >2012-11-08 06:30:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-13) Autorecovering hosts is disabled, skipping >2012-11-08 06:30:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-13) Autorecovering storage domains is disabled, skipping >2012-11-08 06:30:51,820 INFO [org.ovirt.engine.core.bll.storage.RemoveStorageDomainCommand] (ajp-/127.0.0.1:8702-4) [7451f1b8] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e051d09-1633-4510-b6c4-76cdb371a3c6 value: STORAGE >, sharedLocks= ] >2012-11-08 06:30:51,827 INFO [org.ovirt.engine.core.bll.storage.RemoveStorageDomainCommand] (ajp-/127.0.0.1:8702-4) [7451f1b8] Running command: RemoveStorageDomainCommand internal: false. Entities affected : ID: 5e051d09-1633-4510-b6c4-76cdb371a3c6 Type: Storage >2012-11-08 06:30:51,889 INFO [org.ovirt.engine.core.bll.storage.RemoveStorageDomainCommand] (ajp-/127.0.0.1:8702-4) [7451f1b8] Lock freed to object EngineLock [exclusiveLocks= key: 5e051d09-1633-4510-b6c4-76cdb371a3c6 value: STORAGE >, sharedLocks= ] >2012-11-08 06:33:09,824 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-60) Failed to decrypt Data must not be longer than 256 bytes >2012-11-08 06:33:09,838 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-58) Failed to decrypt Data must start with zero >2012-11-08 06:33:24,101 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-82) Failed to decrypt Data must not be longer than 256 bytes >2012-11-08 06:33:24,101 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-84) Failed to decrypt Data must not be longer than 256 bytes >2012-11-08 06:33:34,294 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-5) Failed to decrypt java.lang.NullPointerException >2012-11-08 06:33:34,305 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-2) Failed to decrypt Data must start with zero >2012-11-08 06:33:48,736 INFO [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (ajp-/127.0.0.1:8702-2) [73b3c384] Lock Acquired to object EngineLock [exclusiveLocks= key: b1 value: VM_NAME >, sharedLocks= ] >2012-11-08 06:33:48,760 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-2) [73b3c384] START, IsValidVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 22d15643 >2012-11-08 06:33:48,760 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-2) [73b3c384] FINISH, IsValidVDSCommand, return: true, log id: 22d15643 >2012-11-08 06:33:48,767 INFO [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (ajp-/127.0.0.1:8702-2) [73b3c384] Running command: AddVmFromScratchCommand internal: false. Entities affected : ID: 6cea47a6-2976-11e2-ad99-52540035ed48 Type: VdsGroups, ID: 6cea47a6-2976-11e2-ad99-52540035ed48 Type: VdsGroups >2012-11-08 06:33:48,768 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-2) [73b3c384] START, IsValidVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6e3d5ad >2012-11-08 06:33:48,769 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-2) [73b3c384] FINISH, IsValidVDSCommand, return: true, log id: 6e3d5ad >2012-11-08 06:33:48,835 INFO [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (ajp-/127.0.0.1:8702-2) [73b3c384] Lock freed to object EngineLock [exclusiveLocks= key: b1 value: VM_NAME >, sharedLocks= ] >2012-11-08 06:34:41,769 INFO [org.ovirt.engine.core.bll.AddDiskCommand] (ajp-/127.0.0.1:8702-13) [35219e9f] Lock Acquired to object EngineLock [exclusiveLocks= key: 67bd2d56-8e0b-4328-bcfd-ea7f4758c885 value: VM_DISK_BOOT >, sharedLocks= key: 67bd2d56-8e0b-4328-bcfd-ea7f4758c885 value: VM >] >2012-11-08 06:34:41,793 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-13) [35219e9f] START, IsValidVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 28b2fad8 >2012-11-08 06:34:41,795 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-13) [35219e9f] FINISH, IsValidVDSCommand, return: true, log id: 28b2fad8 >2012-11-08 06:34:41,805 INFO [org.ovirt.engine.core.bll.AddDiskCommand] (ajp-/127.0.0.1:8702-13) [35219e9f] Running command: AddDiskCommand internal: false. Entities affected : ID: 67bd2d56-8e0b-4328-bcfd-ea7f4758c885 Type: VM, ID: a02e96be-cf4c-4d0f-8368-fd9d4aa659bf Type: Storage >2012-11-08 06:34:41,864 INFO [org.ovirt.engine.core.bll.AddImageFromScratchCommand] (ajp-/127.0.0.1:8702-13) [32bcf420] Running command: AddImageFromScratchCommand internal: true. Entities affected : ID: a02e96be-cf4c-4d0f-8368-fd9d4aa659bf Type: Storage >2012-11-08 06:34:41,915 INFO [org.ovirt.engine.core.bll.AddImageFromScratchCommand] (ajp-/127.0.0.1:8702-13) [32bcf420] Lock freed to object EngineLock [exclusiveLocks= key: 67bd2d56-8e0b-4328-bcfd-ea7f4758c885 value: VM_DISK_BOOT >, sharedLocks= key: 67bd2d56-8e0b-4328-bcfd-ea7f4758c885 value: VM >] >2012-11-08 06:34:41,919 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (ajp-/127.0.0.1:8702-13) [32bcf420] START, CreateImageVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = 3.1, storageDomainId = a02e96be-cf4c-4d0f-8368-fd9d4aa659bf, imageGroupId = 6420baa8-3c3c-4085-91b7-29a19e517029, imageSizeInBytes = 5368709120, volumeFormat = RAW, newImageId = 8ba1d2da-5157-4d94-81a4-6b9c41da82f6, newImageDescription = ), log id: 42d930fe >2012-11-08 06:34:41,919 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (ajp-/127.0.0.1:8702-13) [32bcf420] -- CreateImageVDSCommand::ExecuteIrsBrokerCommand: calling 'createVolume' with two new parameters: description and UUID >2012-11-08 06:34:41,919 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (ajp-/127.0.0.1:8702-13) [32bcf420] -- createVolume parameters: > sdUUID=a02e96be-cf4c-4d0f-8368-fd9d4aa659bf > spUUID=3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 > imgGUID=6420baa8-3c3c-4085-91b7-29a19e517029 > size=5,368,709,120 bytes > volFormat=RAW > volType=Preallocated > volUUID=8ba1d2da-5157-4d94-81a4-6b9c41da82f6 > descr= > srcImgGUID=00000000-0000-0000-0000-000000000000 > srcVolUUID=00000000-0000-0000-0000-000000000000 > >2012-11-08 06:34:41,989 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (ajp-/127.0.0.1:8702-13) [32bcf420] FINISH, CreateImageVDSCommand, return: 8ba1d2da-5157-4d94-81a4-6b9c41da82f6, log id: 42d930fe >2012-11-08 06:34:41,996 INFO [org.ovirt.engine.core.bll.EntityAsyncTask] (ajp-/127.0.0.1:8702-13) [32bcf420] EntityAsyncTask::Adding EntityMultiAsyncTasks object for entity 6420baa8-3c3c-4085-91b7-29a19e517029 >2012-11-08 06:34:41,999 INFO [org.ovirt.engine.core.bll.EntityMultiAsyncTasks] (ajp-/127.0.0.1:8702-13) [32bcf420] EntityMultiAsyncTasks::AttachTask: Attaching task 2ced7c3c-61aa-40f4-91a4-614c18826dda to entity 6420baa8-3c3c-4085-91b7-29a19e517029. >2012-11-08 06:34:42,010 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskUtils] (ajp-/127.0.0.1:8702-13) [32bcf420] Adding task 2ced7c3c-61aa-40f4-91a4-614c18826dda to DataBase >2012-11-08 06:34:42,088 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (ajp-/127.0.0.1:8702-13) [32bcf420] Adding task 2ced7c3c-61aa-40f4-91a4-614c18826dda (Parent Command AddDisk, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters), polling hasn't started yet.. >2012-11-08 06:34:42,109 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (ajp-/127.0.0.1:8702-13) [32bcf420] BaseAsyncTask::StartPollingTask: Starting to poll task 2ced7c3c-61aa-40f4-91a4-614c18826dda. >2012-11-08 06:34:42,109 INFO [org.ovirt.engine.core.bll.EntityMultiAsyncTasks] (ajp-/127.0.0.1:8702-13) [32bcf420] EntityMultiAsyncTasks::StartPollingTask: Current Action Type for entity 6420baa8-3c3c-4085-91b7-29a19e517029 is AddDisk (determined by task 2ced7c3c-61aa-40f4-91a4-614c18826dda) >2012-11-08 06:34:46,563 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-33) Polling and updating Async Tasks: 1 tasks, 1 tasks to poll now >2012-11-08 06:34:46,608 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (QuartzScheduler_Worker-33) SPMAsyncTask::PollTask: Polling task 2ced7c3c-61aa-40f4-91a4-614c18826dda (Parent Command AddDisk, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status running. >2012-11-08 06:34:46,608 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-33) Finished polling Tasks, will poll again in 10 seconds. >2012-11-08 06:34:52,282 INFO [org.ovirt.engine.core.bll.AddVmInterfaceCommand] (ajp-/127.0.0.1:8702-8) [71cb0a21] Running command: AddVmInterfaceCommand internal: false. Entities affected : ID: 67bd2d56-8e0b-4328-bcfd-ea7f4758c885 Type: VM >2012-11-08 06:34:52,283 INFO [org.ovirt.engine.core.bll.MacPoolManager] (ajp-/127.0.0.1:8702-8) [71cb0a21] MacPoolManager::allocateNewMac entered >2012-11-08 06:34:52,283 INFO [org.ovirt.engine.core.bll.MacPoolManager] (ajp-/127.0.0.1:8702-8) [71cb0a21] MacPoolManager::allocateNewMac allocated mac = 00:1a:4a:46:22:b0 >2012-11-08 06:34:52,333 INFO [org.ovirt.engine.core.bll.ActivateDeactivateVmNicCommand] (ajp-/127.0.0.1:8702-8) [70147d66] Running command: ActivateDeactivateVmNicCommand internal: true. Entities affected : ID: 67bd2d56-8e0b-4328-bcfd-ea7f4758c885 Type: VM >2012-11-08 06:35:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-64) Autorecovering hosts is disabled, skipping >2012-11-08 06:35:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-64) Autorecovering storage domains is disabled, skipping >2012-11-08 06:35:02,783 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (ajp-/127.0.0.1:8702-1) [840002f] START, ValidateStorageServerConnectionVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: null, connection: guido.lab.eng.blr.redhat.com:/iso, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 2c676543 >2012-11-08 06:35:02,804 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (ajp-/127.0.0.1:8702-1) [840002f] FINISH, ValidateStorageServerConnectionVDSCommand, return: {00000000-0000-0000-0000-000000000000=0}, log id: 2c676543 >2012-11-08 06:35:02,807 INFO [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-1) [840002f] Running command: AddStorageServerConnectionCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2012-11-08 06:35:02,822 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-1) [840002f] START, ConnectStorageServerVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: null, connection: guido.lab.eng.blr.redhat.com:/iso, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 698869ed >2012-11-08 06:35:03,008 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-1) [840002f] FINISH, ConnectStorageServerVDSCommand, return: {00000000-0000-0000-0000-000000000000=100}, log id: 698869ed >2012-11-08 06:35:03,012 ERROR [org.ovirt.engine.core.bll.storage.NFSStorageHelper] (ajp-/127.0.0.1:8702-1) [840002f] The connection with details guido.lab.eng.blr.redhat.com:/iso failed because of error code 100 and error message is: general exception >2012-11-08 06:35:03,879 WARN [org.ovirt.engine.core.bll.storage.AddNFSStorageDomainCommand] (ajp-/127.0.0.1:8702-2) [2306b70c] CanDoAction of action AddNFSStorageDomain failed. Reasons:VAR__ACTION__ADD,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_STORAGE_CONNECTION_NOT_EXIST >2012-11-08 06:35:04,655 INFO [org.ovirt.engine.core.bll.storage.RemoveStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-9) [641e0a6f] Running command: RemoveStorageServerConnectionCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2012-11-08 06:35:04,666 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-9) [641e0a6f] START, DisconnectStorageServerVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: null, connection: guido.lab.eng.blr.redhat.com:/iso, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 18b2958b >2012-11-08 06:35:06,925 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-9) [641e0a6f] FINISH, DisconnectStorageServerVDSCommand, return: {00000000-0000-0000-0000-000000000000=477}, log id: 18b2958b >2012-11-08 06:35:36,804 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (QuartzScheduler_Worker-28) SPMAsyncTask::PollTask: Polling task 2ced7c3c-61aa-40f4-91a4-614c18826dda (Parent Command AddDisk, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status finished, result 'success'. >2012-11-08 06:35:36,807 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (QuartzScheduler_Worker-28) BaseAsyncTask::OnTaskEndSuccess: Task 2ced7c3c-61aa-40f4-91a4-614c18826dda (Parent Command AddDisk, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended successfully. >2012-11-08 06:35:36,808 INFO [org.ovirt.engine.core.bll.EntityAsyncTask] (QuartzScheduler_Worker-28) EntityAsyncTask::EndActionIfNecessary: All tasks of entity 6420baa8-3c3c-4085-91b7-29a19e517029 has ended -> executing EndAction >2012-11-08 06:35:36,808 INFO [org.ovirt.engine.core.bll.EntityAsyncTask] (QuartzScheduler_Worker-28) EntityAsyncTask::EndAction: Ending action for 1 tasks (entity ID: 6420baa8-3c3c-4085-91b7-29a19e517029): calling EndAction for action type AddDisk. >2012-11-08 06:35:36,809 INFO [org.ovirt.engine.core.bll.EntityAsyncTask] (pool-4-thread-49) EntityAsyncTask::EndCommandAction [within thread]context: Attempting to EndAction AddDisk >2012-11-08 06:35:36,831 INFO [org.ovirt.engine.core.bll.AddDiskCommand] (pool-4-thread-49) [35219e9f] Ending command successfully: org.ovirt.engine.core.bll.AddDiskCommand >2012-11-08 06:35:36,832 INFO [org.ovirt.engine.core.bll.AddImageFromScratchCommand] (pool-4-thread-49) [32bcf420] Ending command successfully: org.ovirt.engine.core.bll.AddImageFromScratchCommand >2012-11-08 06:35:36,847 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (pool-4-thread-49) [32bcf420] START, GetImageInfoVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null, storageDomainId = a02e96be-cf4c-4d0f-8368-fd9d4aa659bf, imageGroupId = 6420baa8-3c3c-4085-91b7-29a19e517029, imageId = 8ba1d2da-5157-4d94-81a4-6b9c41da82f6), log id: 34a1580b >2012-11-08 06:35:36,861 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-29) Failed to decrypt Data must not be longer than 256 bytes >2012-11-08 06:35:36,861 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-32) Failed to decrypt Data must not be longer than 256 bytes >2012-11-08 06:35:36,894 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (pool-4-thread-49) [32bcf420] FINISH, GetImageInfoVDSCommand, return: org.ovirt.engine.core.common.businessentities.DiskImage@5d0bc046, log id: 34a1580b >2012-11-08 06:35:37,008 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UpdateVMVDSCommand] (pool-4-thread-49) START, UpdateVMVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null, storageDomainId = 00000000-0000-0000-0000-000000000000, infoDictionary.size = 1), log id: 58dff040 >2012-11-08 06:35:37,038 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UpdateVMVDSCommand] (pool-4-thread-49) FINISH, UpdateVMVDSCommand, log id: 58dff040 >2012-11-08 06:35:37,048 INFO [org.ovirt.engine.core.bll.EntityAsyncTask] (pool-4-thread-49) EntityAsyncTask::HandleEndActionResult [within thread]: EndAction for action type AddDisk completed, handling the result. >2012-11-08 06:35:37,048 INFO [org.ovirt.engine.core.bll.EntityAsyncTask] (pool-4-thread-49) EntityAsyncTask::HandleEndActionResult [within thread]: EndAction for action type AddDisk succeeded, clearing tasks. >2012-11-08 06:35:37,050 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (pool-4-thread-49) SPMAsyncTask::ClearAsyncTask: Attempting to clear task 2ced7c3c-61aa-40f4-91a4-614c18826dda >2012-11-08 06:35:37,050 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (pool-4-thread-49) START, SPMClearTaskVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null, taskId = 2ced7c3c-61aa-40f4-91a4-614c18826dda), log id: 689905a1 >2012-11-08 06:35:37,067 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (pool-4-thread-49) START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=2ced7c3c-61aa-40f4-91a4-614c18826dda), log id: 7a2cc149 >2012-11-08 06:35:37,096 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (pool-4-thread-49) FINISH, HSMClearTaskVDSCommand, log id: 7a2cc149 >2012-11-08 06:35:37,096 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (pool-4-thread-49) FINISH, SPMClearTaskVDSCommand, log id: 689905a1 >2012-11-08 06:35:37,102 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (pool-4-thread-49) BaseAsyncTask::RemoveTaskFromDB: Removed task 2ced7c3c-61aa-40f4-91a4-614c18826dda from DataBase >2012-11-08 06:35:37,102 INFO [org.ovirt.engine.core.bll.EntityAsyncTask] (pool-4-thread-49) EntityAsyncTask::HandleEndActionResult [within thread]: Removing EntityMultiAsyncTasks object for entity 6420baa8-3c3c-4085-91b7-29a19e517029 >2012-11-08 06:35:38,881 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-34) Failed to decrypt Data must not be longer than 256 bytes >2012-11-08 06:35:38,881 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-31) Failed to decrypt Data must not be longer than 256 bytes >2012-11-08 06:35:55,062 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (ajp-/127.0.0.1:8702-4) [1df88223] START, ValidateStorageServerConnectionVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: null, connection: guido.lab.eng.blr.redhat.com:/iso2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: fd2e4a0 >2012-11-08 06:35:55,104 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (ajp-/127.0.0.1:8702-4) [1df88223] FINISH, ValidateStorageServerConnectionVDSCommand, return: {00000000-0000-0000-0000-000000000000=0}, log id: fd2e4a0 >2012-11-08 06:35:55,108 INFO [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-4) [1df88223] Running command: AddStorageServerConnectionCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2012-11-08 06:35:55,127 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-4) [1df88223] START, ConnectStorageServerVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: null, connection: guido.lab.eng.blr.redhat.com:/iso2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 63831559 >2012-11-08 06:35:55,191 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-4) [1df88223] FINISH, ConnectStorageServerVDSCommand, return: {00000000-0000-0000-0000-000000000000=0}, log id: 63831559 >2012-11-08 06:35:56,351 INFO [org.ovirt.engine.core.bll.storage.AddNFSStorageDomainCommand] (ajp-/127.0.0.1:8702-6) [305492d3] Running command: AddNFSStorageDomainCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2012-11-08 06:35:56,375 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-6) [305492d3] START, CreateStorageDomainVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storageDomain=org.ovirt.engine.core.common.businessentities.storage_domain_static@d8762102, args=guido.lab.eng.blr.redhat.com:/iso2), log id: 166e9083 >2012-11-08 06:35:57,268 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-68) Failed to decrypt Data must not be longer than 256 bytes >2012-11-08 06:35:57,268 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-62) Failed to decrypt Data must not be longer than 256 bytes >2012-11-08 06:35:58,907 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (ajp-/127.0.0.1:8702-6) [305492d3] Command org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 361 >mMessage Storage domain is not empty - requires cleaning: ('guido.lab.eng.blr.redhat.com:/iso2',) > > >2012-11-08 06:35:58,907 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (ajp-/127.0.0.1:8702-6) [305492d3] HostName = rhs-gp-srv11.lab.eng.blr.redhat.com >2012-11-08 06:35:58,907 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (ajp-/127.0.0.1:8702-6) [305492d3] Command CreateStorageDomainVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed in vdscommand to CreateStorageDomainVDS, error = Storage domain is not empty - requires cleaning: ('guido.lab.eng.blr.redhat.com:/iso2',) >2012-11-08 06:35:58,907 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-6) [305492d3] FINISH, CreateStorageDomainVDSCommand, log id: 166e9083 >2012-11-08 06:35:58,908 ERROR [org.ovirt.engine.core.bll.storage.AddNFSStorageDomainCommand] (ajp-/127.0.0.1:8702-6) [305492d3] Command org.ovirt.engine.core.bll.storage.AddNFSStorageDomainCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed in vdscommand to CreateStorageDomainVDS, error = Storage domain is not empty - requires cleaning: ('guido.lab.eng.blr.redhat.com:/iso2',) >2012-11-08 06:35:58,922 INFO [org.ovirt.engine.core.bll.storage.AddNFSStorageDomainCommand] (ajp-/127.0.0.1:8702-6) [305492d3] Command [id=dea09826-5959-41fb-b480-0b1fc6127fb8]: Compensating NEW_ENTITY_ID of org.ovirt.engine.core.common.businessentities.storage_domain_dynamic; snapshot: 9dfb9b22-0863-4f2d-b9e3-8ed2f5873a70. >2012-11-08 06:35:58,925 INFO [org.ovirt.engine.core.bll.storage.AddNFSStorageDomainCommand] (ajp-/127.0.0.1:8702-6) [305492d3] Command [id=dea09826-5959-41fb-b480-0b1fc6127fb8]: Compensating NEW_ENTITY_ID of org.ovirt.engine.core.common.businessentities.storage_domain_static; snapshot: 9dfb9b22-0863-4f2d-b9e3-8ed2f5873a70. >2012-11-08 06:35:58,931 ERROR [org.ovirt.engine.core.bll.storage.AddNFSStorageDomainCommand] (ajp-/127.0.0.1:8702-6) [305492d3] Transaction rolled-back for command: org.ovirt.engine.core.bll.storage.AddNFSStorageDomainCommand. >2012-11-08 06:35:59,484 INFO [org.ovirt.engine.core.bll.storage.RemoveStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-1) [6c5c5b66] Running command: RemoveStorageServerConnectionCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2012-11-08 06:35:59,496 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-1) [6c5c5b66] START, DisconnectStorageServerVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: null, connection: guido.lab.eng.blr.redhat.com:/iso2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 22cd8cd5 >2012-11-08 06:36:01,713 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-1) [6c5c5b66] FINISH, DisconnectStorageServerVDSCommand, return: {00000000-0000-0000-0000-000000000000=0}, log id: 22cd8cd5 >2012-11-08 06:36:19,750 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-6) Failed to decrypt Data must start with zero >2012-11-08 06:36:21,252 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-11) [e0d0678] START, IsValidVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6c8684cf >2012-11-08 06:36:21,252 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-11) [e0d0678] FINISH, IsValidVDSCommand, return: true, log id: 6c8684cf >2012-11-08 06:36:21,264 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp-/127.0.0.1:8702-11) [e0d0678] START, IsVmDuringInitiatingVDSCommand( vmId = 67bd2d56-8e0b-4328-bcfd-ea7f4758c885), log id: 194140bf >2012-11-08 06:36:21,264 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp-/127.0.0.1:8702-11) [e0d0678] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 194140bf >2012-11-08 06:36:21,311 INFO [org.ovirt.engine.core.bll.RunVmOnceCommand] (ajp-/127.0.0.1:8702-11) [e0d0678] Running command: RunVmOnceCommand internal: false. Entities affected : ID: 67bd2d56-8e0b-4328-bcfd-ea7f4758c885 Type: VM, ID: 67bd2d56-8e0b-4328-bcfd-ea7f4758c885 Type: VM >2012-11-08 06:36:21,397 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-11) [e0d0678] START, CreateVmVDSCommand(HostName = rhs-gp-srv4.lab.eng.blr.redhat.com, HostId = d2bd32d4-2992-11e2-ab59-52540035ed48, vmId=67bd2d56-8e0b-4328-bcfd-ea7f4758c885, vm=org.ovirt.engine.core.common.businessentities.VM@55e80005), log id: 344c146f >2012-11-08 06:36:21,427 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8702-11) [e0d0678] START, CreateVDSCommand(HostName = rhs-gp-srv4.lab.eng.blr.redhat.com, HostId = d2bd32d4-2992-11e2-ab59-52540035ed48, vmId=67bd2d56-8e0b-4328-bcfd-ea7f4758c885, vm=org.ovirt.engine.core.common.businessentities.VM@55e80005), log id: 565d5ea6 >2012-11-08 06:36:21,509 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8702-11) [e0d0678] org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand spiceSslCipherSuite=DEFAULT,memSize=512,kvmEnable=true,smp=1,emulatedMachine=rhel6.3.0,vmType=kvm,keyboardLayout=en-us,nice=0,display=qxl,smpCoresPerSocket=1,spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard,timeOffset=-43200,transparentHugePages=true,vmId=67bd2d56-8e0b-4328-bcfd-ea7f4758c885,devices=[Ljava.util.Map;@cc1a1d3,acpiEnable=true,vmName=b1,cpuType=Nehalem,custom={viodiskcache=none} >2012-11-08 06:36:21,510 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8702-11) [e0d0678] FINISH, CreateVDSCommand, log id: 565d5ea6 >2012-11-08 06:36:21,512 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-11) [e0d0678] IncreasePendingVms::CreateVmIncreasing vds rhs-gp-srv4.lab.eng.blr.redhat.com pending vcpu count, now 1. Vm: b1 >2012-11-08 06:36:21,519 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-11) [e0d0678] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 344c146f >2012-11-08 06:36:21,849 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (QuartzScheduler_Worker-14) START, FullListVdsCommand( HostId = d2bd32d4-2992-11e2-ab59-52540035ed48, vds=null, vmIds=[67bd2d56-8e0b-4328-bcfd-ea7f4758c885]), log id: 474f8a61 >2012-11-08 06:36:21,866 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (QuartzScheduler_Worker-14) FINISH, FullListVdsCommand, return: [Lorg.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcStruct;@37060ada, log id: 474f8a61 >2012-11-08 06:36:21,876 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-14) Recieved a qxl Device without an address when processing VM 67bd2d56-8e0b-4328-bcfd-ea7f4758c885 devices, skipping device: {specParams={vram=65536}, device=qxl, type=video, deviceId=4fc2bde3-f994-4d90-9afe-31d99749cb50} >2012-11-08 06:36:21,876 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-14) Recieved a bridge Device without an address when processing VM 67bd2d56-8e0b-4328-bcfd-ea7f4758c885 devices, skipping device: {bootOrder=2, nicModel=pv, specParams={}, device=bridge, macAddr=00:1a:4a:46:22:b0, type=interface, deviceId=6f2e9a31-58f7-4fcc-ba7d-a9157c6d95bf, filter=vdsm-no-mac-spoofing, network=rhevm} >2012-11-08 06:36:21,876 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-14) Recieved a memballoon Device without an address when processing VM 67bd2d56-8e0b-4328-bcfd-ea7f4758c885 devices, skipping device: {specParams={model=virtio}, device=memballoon, type=balloon, deviceId=97ac78de-2de8-4bb6-9059-d0d3839ad665} >2012-11-08 06:36:21,876 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-14) Recieved a cdrom Device without an address when processing VM 67bd2d56-8e0b-4328-bcfd-ea7f4758c885 devices, skipping device: {shared=false, iface=ide, index=2, specParams={path=}, device=cdrom, path=, type=disk, readonly=true, deviceId=125f9d54-e547-4f87-83e6-fb5752adfab9} >2012-11-08 06:36:21,876 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-14) Recieved a disk Device without an address when processing VM 67bd2d56-8e0b-4328-bcfd-ea7f4758c885 devices, skipping device: {shared=false, index=0, volumeID=8ba1d2da-5157-4d94-81a4-6b9c41da82f6, propagateErrors=off, format=raw, type=disk, truesize=5368709120, reqsize=0, iface=virtio, bootOrder=1, volumeChain=[Ljava.lang.Object;@78de9074, imageID=6420baa8-3c3c-4085-91b7-29a19e517029, domainID=a02e96be-cf4c-4d0f-8368-fd9d4aa659bf, specParams={}, optional=false, path=/rhev/data-center/3a17dcac-a6e3-4563-a4b0-f27d89b68dd6/a02e96be-cf4c-4d0f-8368-fd9d4aa659bf/images/6420baa8-3c3c-4085-91b7-29a19e517029/8ba1d2da-5157-4d94-81a4-6b9c41da82f6, device=disk, poolID=3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, readonly=false, deviceId=6420baa8-3c3c-4085-91b7-29a19e517029, apparentsize=5368709120} >2012-11-08 06:36:23,977 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (QuartzScheduler_Worker-15) START, DestroyVDSCommand(HostName = rhs-gp-srv4.lab.eng.blr.redhat.com, HostId = d2bd32d4-2992-11e2-ab59-52540035ed48, vmId=67bd2d56-8e0b-4328-bcfd-ea7f4758c885, force=false, secondsToWait=0, gracefully=false), log id: 5d24b5ee >2012-11-08 06:36:24,037 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (QuartzScheduler_Worker-15) FINISH, DestroyVDSCommand, log id: 5d24b5ee >2012-11-08 06:36:24,055 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-15) Running on vds during rerun failed vm: null >2012-11-08 06:36:24,056 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-15) vm b1 running in db and not running in vds - add to rerun treatment. vds rhs-gp-srv4.lab.eng.blr.redhat.com >2012-11-08 06:36:24,060 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-15) Rerun vm 67bd2d56-8e0b-4328-bcfd-ea7f4758c885. Called from vds rhs-gp-srv4.lab.eng.blr.redhat.com >2012-11-08 06:36:24,082 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsDynamicDataVDSCommand] (pool-4-thread-50) START, UpdateVdsDynamicDataVDSCommand(HostName = rhs-gp-srv4.lab.eng.blr.redhat.com, HostId = d2bd32d4-2992-11e2-ab59-52540035ed48, vdsDynamic=org.ovirt.engine.core.common.businessentities.VdsDynamic@725ab7ae), log id: 2d71c41d >2012-11-08 06:36:24,084 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsDynamicDataVDSCommand] (pool-4-thread-50) FINISH, UpdateVdsDynamicDataVDSCommand, log id: 2d71c41d >2012-11-08 06:36:24,126 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (pool-4-thread-50) START, IsValidVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: ea0b46e >2012-11-08 06:36:24,127 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (pool-4-thread-50) FINISH, IsValidVDSCommand, return: true, log id: ea0b46e >2012-11-08 06:36:24,138 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (pool-4-thread-50) START, IsVmDuringInitiatingVDSCommand( vmId = 67bd2d56-8e0b-4328-bcfd-ea7f4758c885), log id: 2f4c2339 >2012-11-08 06:36:24,138 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (pool-4-thread-50) FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 2f4c2339 >2012-11-08 06:36:24,182 INFO [org.ovirt.engine.core.bll.RunVmOnceCommand] (pool-4-thread-50) Running command: RunVmOnceCommand internal: false. Entities affected : ID: 67bd2d56-8e0b-4328-bcfd-ea7f4758c885 Type: VM, ID: 67bd2d56-8e0b-4328-bcfd-ea7f4758c885 Type: VM >2012-11-08 06:36:24,259 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (pool-4-thread-50) START, CreateVmVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=67bd2d56-8e0b-4328-bcfd-ea7f4758c885, vm=org.ovirt.engine.core.common.businessentities.VM@404d8b82), log id: 14e1de1b >2012-11-08 06:36:24,273 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (pool-4-thread-50) START, CreateVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=67bd2d56-8e0b-4328-bcfd-ea7f4758c885, vm=org.ovirt.engine.core.common.businessentities.VM@404d8b82), log id: 4883fe39 >2012-11-08 06:36:24,340 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (pool-4-thread-50) org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand spiceSslCipherSuite=DEFAULT,memSize=512,kvmEnable=true,smp=1,emulatedMachine=rhel6.3.0,vmType=kvm,keyboardLayout=en-us,nice=0,display=qxl,smpCoresPerSocket=1,spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard,timeOffset=-43200,transparentHugePages=true,vmId=67bd2d56-8e0b-4328-bcfd-ea7f4758c885,devices=[Ljava.util.Map;@1d1dce46,acpiEnable=true,vmName=b1,cpuType=Nehalem,custom={viodiskcache=none} >2012-11-08 06:36:24,340 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (pool-4-thread-50) FINISH, CreateVDSCommand, log id: 4883fe39 >2012-11-08 06:36:24,340 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (pool-4-thread-50) IncreasePendingVms::CreateVmIncreasing vds rhs-gp-srv11.lab.eng.blr.redhat.com pending vcpu count, now 1. Vm: b1 >2012-11-08 06:36:24,343 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (pool-4-thread-50) FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 14e1de1b >2012-11-08 06:36:25,976 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (QuartzScheduler_Worker-23) START, DestroyVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=67bd2d56-8e0b-4328-bcfd-ea7f4758c885, force=false, secondsToWait=0, gracefully=false), log id: 4c7765ea >2012-11-08 06:36:26,038 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (QuartzScheduler_Worker-23) FINISH, DestroyVDSCommand, log id: 4c7765ea >2012-11-08 06:36:26,057 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-23) Running on vds during rerun failed vm: null >2012-11-08 06:36:26,058 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-23) vm b1 running in db and not running in vds - add to rerun treatment. vds rhs-gp-srv11.lab.eng.blr.redhat.com >2012-11-08 06:36:26,063 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-23) Rerun vm 67bd2d56-8e0b-4328-bcfd-ea7f4758c885. Called from vds rhs-gp-srv11.lab.eng.blr.redhat.com >2012-11-08 06:36:26,101 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsDynamicDataVDSCommand] (pool-4-thread-50) START, UpdateVdsDynamicDataVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vdsDynamic=org.ovirt.engine.core.common.businessentities.VdsDynamic@e45c6897), log id: 5773b859 >2012-11-08 06:36:26,103 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsDynamicDataVDSCommand] (pool-4-thread-50) FINISH, UpdateVdsDynamicDataVDSCommand, log id: 5773b859 >2012-11-08 06:36:26,134 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (pool-4-thread-50) START, IsValidVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 66fea8a8 >2012-11-08 06:36:26,135 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (pool-4-thread-50) FINISH, IsValidVDSCommand, return: true, log id: 66fea8a8 >2012-11-08 06:36:26,145 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (pool-4-thread-50) START, IsVmDuringInitiatingVDSCommand( vmId = 67bd2d56-8e0b-4328-bcfd-ea7f4758c885), log id: 79201512 >2012-11-08 06:36:26,145 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (pool-4-thread-50) FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 79201512 >2012-11-08 06:36:26,192 INFO [org.ovirt.engine.core.bll.VdsSelector] (pool-4-thread-50) VDS rhs-gp-srv11.lab.eng.blr.redhat.com 96739be4-2990-11e2-8577-52540035ed48 have failed running this VM in the current selection cycle VDS rhs-gp-srv4.lab.eng.blr.redhat.com d2bd32d4-2992-11e2-ab59-52540035ed48 is the same host the VM is currently running on VDS rhs-client11.lab.eng.blr.redhat.com 90b581c2-2977-11e2-82cd-52540035ed48 is not in up status or belongs to the VM's cluster >2012-11-08 06:36:26,193 WARN [org.ovirt.engine.core.bll.RunVmOnceCommand] (pool-4-thread-50) CanDoAction of action RunVmOnce failed. Reasons:VAR__ACTION__RUN,VAR__TYPE__VM,$VmName b1,VAR__ACTION__RUN,VAR__TYPE__VM,$VmName b1,VAR__ACTION__RUN,VAR__TYPE__VM,ACTION_TYPE_FAILED_VDS_VM_CLUSTER >2012-11-08 06:36:36,273 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-37) Failed to decrypt Data must not be longer than 256 bytes >2012-11-08 06:36:36,273 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-41) Failed to decrypt Data must not be longer than 256 bytes >2012-11-08 06:36:46,366 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (ajp-/127.0.0.1:8702-5) [5b6b7351] START, ValidateStorageServerConnectionVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: null, connection: guido.lab.eng.blr.redhat.com:/iso2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 4fa6bc09 >2012-11-08 06:36:46,384 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (ajp-/127.0.0.1:8702-5) [5b6b7351] FINISH, ValidateStorageServerConnectionVDSCommand, return: {00000000-0000-0000-0000-000000000000=0}, log id: 4fa6bc09 >2012-11-08 06:36:46,390 INFO [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-5) [5b6b7351] Running command: AddStorageServerConnectionCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2012-11-08 06:36:46,406 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-5) [5b6b7351] START, ConnectStorageServerVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 54270a93-f87f-477e-ad6d-e001aa34dae0, connection: guido.lab.eng.blr.redhat.com:/iso2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 3be0fe2b >2012-11-08 06:36:46,480 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-5) [5b6b7351] FINISH, ConnectStorageServerVDSCommand, return: {54270a93-f87f-477e-ad6d-e001aa34dae0=0}, log id: 3be0fe2b >2012-11-08 06:36:47,675 INFO [org.ovirt.engine.core.bll.storage.AddNFSStorageDomainCommand] (ajp-/127.0.0.1:8702-4) [3d974062] Running command: AddNFSStorageDomainCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2012-11-08 06:36:47,694 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-4) [3d974062] START, CreateStorageDomainVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storageDomain=org.ovirt.engine.core.common.businessentities.storage_domain_static@740a3da4, args=guido.lab.eng.blr.redhat.com:/iso2), log id: 4a8be44e >2012-11-08 06:36:50,239 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-4) [3d974062] FINISH, CreateStorageDomainVDSCommand, log id: 4a8be44e >2012-11-08 06:36:50,256 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStorageDomainStatsVDSCommand] (ajp-/127.0.0.1:8702-4) [3d974062] START, GetStorageDomainStatsVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storageDomainId=31a63154-f9f3-45a9-b85e-eeca80451c89), log id: 20049703 >2012-11-08 06:36:50,278 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStorageDomainStatsVDSCommand] (ajp-/127.0.0.1:8702-4) [3d974062] FINISH, GetStorageDomainStatsVDSCommand, return: org.ovirt.engine.core.common.businessentities.storage_domains@fe3b2841, log id: 20049703 >2012-11-08 06:36:50,973 INFO [org.ovirt.engine.core.bll.storage.RemoveStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-10) [b3b6c40] Running command: RemoveStorageServerConnectionCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2012-11-08 06:36:50,986 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-10) [b3b6c40] START, DisconnectStorageServerVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: null, connection: guido.lab.eng.blr.redhat.com:/iso2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 1496cb5f >2012-11-08 06:36:53,203 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-10) [b3b6c40] FINISH, DisconnectStorageServerVDSCommand, return: {00000000-0000-0000-0000-000000000000=0}, log id: 1496cb5f >2012-11-08 06:36:56,211 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-74) Setting new tasks map. The map contains now 0 tasks >2012-11-08 06:36:56,211 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-74) Cleared all tasks of pool 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6. >2012-11-08 06:36:56,236 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-74) Could not find vds that is spm and non-operational. >2012-11-08 06:37:23,849 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-9) [2b76af87] START, IsValidVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 34d88664 >2012-11-08 06:37:23,850 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-9) [2b76af87] FINISH, IsValidVDSCommand, return: true, log id: 34d88664 >2012-11-08 06:37:23,856 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp-/127.0.0.1:8702-9) [2b76af87] START, IsVmDuringInitiatingVDSCommand( vmId = 67bd2d56-8e0b-4328-bcfd-ea7f4758c885), log id: caf0038 >2012-11-08 06:37:23,856 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp-/127.0.0.1:8702-9) [2b76af87] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: caf0038 >2012-11-08 06:37:23,897 INFO [org.ovirt.engine.core.bll.RunVmOnceCommand] (ajp-/127.0.0.1:8702-9) [2b76af87] Running command: RunVmOnceCommand internal: false. Entities affected : ID: 67bd2d56-8e0b-4328-bcfd-ea7f4758c885 Type: VM, ID: 67bd2d56-8e0b-4328-bcfd-ea7f4758c885 Type: VM >2012-11-08 06:37:23,952 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-9) [2b76af87] START, CreateVmVDSCommand(HostName = rhs-gp-srv4.lab.eng.blr.redhat.com, HostId = d2bd32d4-2992-11e2-ab59-52540035ed48, vmId=67bd2d56-8e0b-4328-bcfd-ea7f4758c885, vm=org.ovirt.engine.core.common.businessentities.VM@4cf3e196), log id: 4a406668 >2012-11-08 06:37:23,965 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8702-9) [2b76af87] START, CreateVDSCommand(HostName = rhs-gp-srv4.lab.eng.blr.redhat.com, HostId = d2bd32d4-2992-11e2-ab59-52540035ed48, vmId=67bd2d56-8e0b-4328-bcfd-ea7f4758c885, vm=org.ovirt.engine.core.common.businessentities.VM@4cf3e196), log id: 7ae688e6 >2012-11-08 06:37:24,008 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8702-9) [2b76af87] org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand spiceSslCipherSuite=DEFAULT,memSize=512,kvmEnable=true,smp=1,emulatedMachine=rhel6.3.0,vmType=kvm,keyboardLayout=en-us,nice=0,display=qxl,smpCoresPerSocket=1,spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard,timeOffset=-43200,transparentHugePages=true,vmId=67bd2d56-8e0b-4328-bcfd-ea7f4758c885,devices=[Ljava.util.Map;@39db70f9,acpiEnable=true,vmName=b1,cpuType=Nehalem,custom={viodiskcache=none} >2012-11-08 06:37:24,008 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8702-9) [2b76af87] FINISH, CreateVDSCommand, log id: 7ae688e6 >2012-11-08 06:37:24,009 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-9) [2b76af87] IncreasePendingVms::CreateVmIncreasing vds rhs-gp-srv4.lab.eng.blr.redhat.com pending vcpu count, now 1. Vm: b1 >2012-11-08 06:37:24,011 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-9) [2b76af87] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 4a406668 >2012-11-08 06:37:25,420 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (QuartzScheduler_Worker-9) START, DestroyVDSCommand(HostName = rhs-gp-srv4.lab.eng.blr.redhat.com, HostId = d2bd32d4-2992-11e2-ab59-52540035ed48, vmId=67bd2d56-8e0b-4328-bcfd-ea7f4758c885, force=false, secondsToWait=0, gracefully=false), log id: 1078b912 >2012-11-08 06:37:25,476 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (QuartzScheduler_Worker-9) FINISH, DestroyVDSCommand, log id: 1078b912 >2012-11-08 06:37:25,494 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-9) Running on vds during rerun failed vm: null >2012-11-08 06:37:25,495 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-9) vm b1 running in db and not running in vds - add to rerun treatment. vds rhs-gp-srv4.lab.eng.blr.redhat.com >2012-11-08 06:37:25,498 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-9) Rerun vm 67bd2d56-8e0b-4328-bcfd-ea7f4758c885. Called from vds rhs-gp-srv4.lab.eng.blr.redhat.com >2012-11-08 06:37:25,518 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsDynamicDataVDSCommand] (pool-4-thread-47) START, UpdateVdsDynamicDataVDSCommand(HostName = rhs-gp-srv4.lab.eng.blr.redhat.com, HostId = d2bd32d4-2992-11e2-ab59-52540035ed48, vdsDynamic=org.ovirt.engine.core.common.businessentities.VdsDynamic@725ab7ae), log id: 5dca35dc >2012-11-08 06:37:25,520 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsDynamicDataVDSCommand] (pool-4-thread-47) FINISH, UpdateVdsDynamicDataVDSCommand, log id: 5dca35dc >2012-11-08 06:37:25,550 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (pool-4-thread-47) START, IsValidVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 1431cf62 >2012-11-08 06:37:25,551 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (pool-4-thread-47) FINISH, IsValidVDSCommand, return: true, log id: 1431cf62 >2012-11-08 06:37:25,564 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (pool-4-thread-47) START, IsVmDuringInitiatingVDSCommand( vmId = 67bd2d56-8e0b-4328-bcfd-ea7f4758c885), log id: 55e117c6 >2012-11-08 06:37:25,564 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (pool-4-thread-47) FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 55e117c6 >2012-11-08 06:37:25,600 INFO [org.ovirt.engine.core.bll.RunVmOnceCommand] (pool-4-thread-47) Running command: RunVmOnceCommand internal: false. Entities affected : ID: 67bd2d56-8e0b-4328-bcfd-ea7f4758c885 Type: VM, ID: 67bd2d56-8e0b-4328-bcfd-ea7f4758c885 Type: VM >2012-11-08 06:37:25,662 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (pool-4-thread-47) START, CreateVmVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=67bd2d56-8e0b-4328-bcfd-ea7f4758c885, vm=org.ovirt.engine.core.common.businessentities.VM@7125e1f3), log id: 6c99d056 >2012-11-08 06:37:25,672 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (pool-4-thread-47) START, CreateVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=67bd2d56-8e0b-4328-bcfd-ea7f4758c885, vm=org.ovirt.engine.core.common.businessentities.VM@7125e1f3), log id: 5a40530d >2012-11-08 06:37:25,722 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (pool-4-thread-47) org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand spiceSslCipherSuite=DEFAULT,memSize=512,kvmEnable=true,smp=1,emulatedMachine=rhel6.3.0,vmType=kvm,keyboardLayout=en-us,nice=0,display=qxl,smpCoresPerSocket=1,spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard,timeOffset=-43200,transparentHugePages=true,vmId=67bd2d56-8e0b-4328-bcfd-ea7f4758c885,devices=[Ljava.util.Map;@68cfc0f6,acpiEnable=true,vmName=b1,cpuType=Nehalem,custom={viodiskcache=none} >2012-11-08 06:37:25,722 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (pool-4-thread-47) FINISH, CreateVDSCommand, log id: 5a40530d >2012-11-08 06:37:25,722 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (pool-4-thread-47) IncreasePendingVms::CreateVmIncreasing vds rhs-gp-srv11.lab.eng.blr.redhat.com pending vcpu count, now 1. Vm: b1 >2012-11-08 06:37:25,724 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (pool-4-thread-47) FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 6c99d056 >2012-11-08 06:37:27,447 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (QuartzScheduler_Worker-42) START, DestroyVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=67bd2d56-8e0b-4328-bcfd-ea7f4758c885, force=false, secondsToWait=0, gracefully=false), log id: 4e6fdf2f >2012-11-08 06:37:27,502 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (QuartzScheduler_Worker-42) FINISH, DestroyVDSCommand, log id: 4e6fdf2f >2012-11-08 06:37:27,552 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-42) Running on vds during rerun failed vm: null >2012-11-08 06:37:27,553 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-42) vm b1 running in db and not running in vds - add to rerun treatment. vds rhs-gp-srv11.lab.eng.blr.redhat.com >2012-11-08 06:37:27,556 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-42) Rerun vm 67bd2d56-8e0b-4328-bcfd-ea7f4758c885. Called from vds rhs-gp-srv11.lab.eng.blr.redhat.com >2012-11-08 06:37:27,577 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsDynamicDataVDSCommand] (pool-4-thread-47) START, UpdateVdsDynamicDataVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vdsDynamic=org.ovirt.engine.core.common.businessentities.VdsDynamic@a701a1d5), log id: 29f7d099 >2012-11-08 06:37:27,578 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsDynamicDataVDSCommand] (pool-4-thread-47) FINISH, UpdateVdsDynamicDataVDSCommand, log id: 29f7d099 >2012-11-08 06:37:27,620 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (pool-4-thread-47) START, IsValidVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 13d9c012 >2012-11-08 06:37:27,621 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (pool-4-thread-47) FINISH, IsValidVDSCommand, return: true, log id: 13d9c012 >2012-11-08 06:37:27,627 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (pool-4-thread-47) START, IsVmDuringInitiatingVDSCommand( vmId = 67bd2d56-8e0b-4328-bcfd-ea7f4758c885), log id: 5b94eda3 >2012-11-08 06:37:27,627 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (pool-4-thread-47) FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 5b94eda3 >2012-11-08 06:37:27,664 INFO [org.ovirt.engine.core.bll.VdsSelector] (pool-4-thread-47) VDS rhs-gp-srv11.lab.eng.blr.redhat.com 96739be4-2990-11e2-8577-52540035ed48 have failed running this VM in the current selection cycle VDS rhs-gp-srv4.lab.eng.blr.redhat.com d2bd32d4-2992-11e2-ab59-52540035ed48 is the same host the VM is currently running on VDS rhs-client11.lab.eng.blr.redhat.com 90b581c2-2977-11e2-82cd-52540035ed48 is not in up status or belongs to the VM's cluster >2012-11-08 06:37:27,665 WARN [org.ovirt.engine.core.bll.RunVmOnceCommand] (pool-4-thread-47) CanDoAction of action RunVmOnce failed. Reasons:VAR__ACTION__RUN,VAR__TYPE__VM,$VmName b1,VAR__ACTION__RUN,VAR__TYPE__VM,$VmName b1,VAR__ACTION__RUN,VAR__TYPE__VM,ACTION_TYPE_FAILED_VDS_VM_CLUSTER >2012-11-08 06:40:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-19) Autorecovering hosts is disabled, skipping >2012-11-08 06:40:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-19) Autorecovering storage domains is disabled, skipping >2012-11-08 06:40:31,400 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-73) Failed to decrypt Data must not be longer than 256 bytes >2012-11-08 06:40:31,400 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-77) Failed to decrypt Data must not be longer than 256 bytes >2012-11-08 06:40:36,021 INFO [org.ovirt.engine.core.bll.storage.RemoveStorageDomainCommand] (ajp-/127.0.0.1:8702-7) [39b4a0fd] Lock Acquired to object EngineLock [exclusiveLocks= key: 31a63154-f9f3-45a9-b85e-eeca80451c89 value: STORAGE >, sharedLocks= ] >2012-11-08 06:40:36,028 INFO [org.ovirt.engine.core.bll.storage.RemoveStorageDomainCommand] (ajp-/127.0.0.1:8702-7) [39b4a0fd] Running command: RemoveStorageDomainCommand internal: false. Entities affected : ID: 31a63154-f9f3-45a9-b85e-eeca80451c89 Type: Storage >2012-11-08 06:40:36,057 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (ajp-/127.0.0.1:8702-7) [2ef41bd2] START, ValidateStorageServerConnectionVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: f7356129-c010-44e9-a186-a500888b0ac5, connection: guido.lab.eng.blr.redhat.com:/iso2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 7982f144 >2012-11-08 06:40:36,082 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (ajp-/127.0.0.1:8702-7) [2ef41bd2] FINISH, ValidateStorageServerConnectionVDSCommand, return: {f7356129-c010-44e9-a186-a500888b0ac5=0}, log id: 7982f144 >2012-11-08 06:40:36,082 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (ajp-/127.0.0.1:8702-7) [2ef41bd2] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2012-11-08 06:40:36,097 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-7) [2ef41bd2] START, ConnectStorageServerVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: f7356129-c010-44e9-a186-a500888b0ac5, connection: guido.lab.eng.blr.redhat.com:/iso2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 26b82d52 >2012-11-08 06:40:36,167 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-7) [2ef41bd2] FINISH, ConnectStorageServerVDSCommand, return: {f7356129-c010-44e9-a186-a500888b0ac5=0}, log id: 26b82d52 >2012-11-08 06:40:36,180 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FormatStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-7) [2ef41bd2] START, FormatStorageDomainVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storageDomainId=31a63154-f9f3-45a9-b85e-eeca80451c89), log id: 72c122b9 >2012-11-08 06:40:40,831 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FormatStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-7) [2ef41bd2] FINISH, FormatStorageDomainVDSCommand, log id: 72c122b9 >2012-11-08 06:40:40,831 WARN [org.ovirt.engine.core.dal.dbbroker.generic.DBConfigUtils] (ajp-/127.0.0.1:8702-7) [2ef41bd2] Didnt find the value of IsNeedSupportForOldVgAPI in DB for version 3.1 - using default: false >2012-11-08 06:40:40,833 INFO [org.ovirt.engine.core.bll.storage.RemoveStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-7) [4d456b3e] Running command: RemoveStorageServerConnectionCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2012-11-08 06:40:40,848 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-7) [4d456b3e] START, DisconnectStorageServerVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: f7356129-c010-44e9-a186-a500888b0ac5, connection: guido.lab.eng.blr.redhat.com:/iso2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 12907dcc >2012-11-08 06:40:43,064 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-7) [4d456b3e] FINISH, DisconnectStorageServerVDSCommand, return: {f7356129-c010-44e9-a186-a500888b0ac5=0}, log id: 12907dcc >2012-11-08 06:40:43,073 INFO [org.ovirt.engine.core.bll.storage.RemoveStorageDomainCommand] (ajp-/127.0.0.1:8702-7) [4d456b3e] Lock freed to object EngineLock [exclusiveLocks= key: 31a63154-f9f3-45a9-b85e-eeca80451c89 value: STORAGE >, sharedLocks= ] >2012-11-08 06:42:46,267 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-19) Failed to decrypt Data must not be longer than 256 bytes >2012-11-08 06:42:46,267 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-22) Failed to decrypt Data must not be longer than 256 bytes >2012-11-08 06:43:03,008 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-3) [11b63371] START, IsValidVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 8a2c7b8 >2012-11-08 06:43:03,008 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-3) [11b63371] FINISH, IsValidVDSCommand, return: true, log id: 8a2c7b8 >2012-11-08 06:43:03,015 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp-/127.0.0.1:8702-3) [11b63371] START, IsVmDuringInitiatingVDSCommand( vmId = 67bd2d56-8e0b-4328-bcfd-ea7f4758c885), log id: 2c5babfa >2012-11-08 06:43:03,015 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp-/127.0.0.1:8702-3) [11b63371] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 2c5babfa >2012-11-08 06:43:03,063 INFO [org.ovirt.engine.core.bll.RunVmOnceCommand] (ajp-/127.0.0.1:8702-3) [11b63371] Running command: RunVmOnceCommand internal: false. Entities affected : ID: 67bd2d56-8e0b-4328-bcfd-ea7f4758c885 Type: VM, ID: 67bd2d56-8e0b-4328-bcfd-ea7f4758c885 Type: VM >2012-11-08 06:43:03,138 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-3) [11b63371] START, CreateVmVDSCommand(HostName = rhs-gp-srv4.lab.eng.blr.redhat.com, HostId = d2bd32d4-2992-11e2-ab59-52540035ed48, vmId=67bd2d56-8e0b-4328-bcfd-ea7f4758c885, vm=org.ovirt.engine.core.common.businessentities.VM@5adc6db5), log id: 365db34e >2012-11-08 06:43:03,158 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8702-3) [11b63371] START, CreateVDSCommand(HostName = rhs-gp-srv4.lab.eng.blr.redhat.com, HostId = d2bd32d4-2992-11e2-ab59-52540035ed48, vmId=67bd2d56-8e0b-4328-bcfd-ea7f4758c885, vm=org.ovirt.engine.core.common.businessentities.VM@5adc6db5), log id: 5e52eea >2012-11-08 06:43:03,205 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8702-3) [11b63371] org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand spiceSslCipherSuite=DEFAULT,memSize=512,kvmEnable=true,smp=1,emulatedMachine=rhel6.3.0,vmType=kvm,keyboardLayout=en-us,nice=0,display=qxl,smpCoresPerSocket=1,spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard,timeOffset=-43200,transparentHugePages=true,vmId=67bd2d56-8e0b-4328-bcfd-ea7f4758c885,devices=[Ljava.util.Map;@28260493,acpiEnable=true,vmName=b1,cpuType=Nehalem,custom={viodiskcache=none} >2012-11-08 06:43:03,205 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8702-3) [11b63371] FINISH, CreateVDSCommand, log id: 5e52eea >2012-11-08 06:43:03,206 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-3) [11b63371] IncreasePendingVms::CreateVmIncreasing vds rhs-gp-srv4.lab.eng.blr.redhat.com pending vcpu count, now 1. Vm: b1 >2012-11-08 06:43:03,208 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-3) [11b63371] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 365db34e >2012-11-08 06:43:04,698 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (QuartzScheduler_Worker-58) START, DestroyVDSCommand(HostName = rhs-gp-srv4.lab.eng.blr.redhat.com, HostId = d2bd32d4-2992-11e2-ab59-52540035ed48, vmId=67bd2d56-8e0b-4328-bcfd-ea7f4758c885, force=false, secondsToWait=0, gracefully=false), log id: 6fbcead2 >2012-11-08 06:43:04,754 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (QuartzScheduler_Worker-58) FINISH, DestroyVDSCommand, log id: 6fbcead2 >2012-11-08 06:43:04,787 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-58) Running on vds during rerun failed vm: null >2012-11-08 06:43:04,788 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-58) vm b1 running in db and not running in vds - add to rerun treatment. vds rhs-gp-srv4.lab.eng.blr.redhat.com >2012-11-08 06:43:04,793 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-58) Rerun vm 67bd2d56-8e0b-4328-bcfd-ea7f4758c885. Called from vds rhs-gp-srv4.lab.eng.blr.redhat.com >2012-11-08 06:43:04,820 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsDynamicDataVDSCommand] (pool-4-thread-49) START, UpdateVdsDynamicDataVDSCommand(HostName = rhs-gp-srv4.lab.eng.blr.redhat.com, HostId = d2bd32d4-2992-11e2-ab59-52540035ed48, vdsDynamic=org.ovirt.engine.core.common.businessentities.VdsDynamic@34fff0ec), log id: f8849c8 >2012-11-08 06:43:04,821 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsDynamicDataVDSCommand] (pool-4-thread-49) FINISH, UpdateVdsDynamicDataVDSCommand, log id: f8849c8 >2012-11-08 06:43:04,845 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (pool-4-thread-49) START, IsValidVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: f3b6397 >2012-11-08 06:43:04,846 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (pool-4-thread-49) FINISH, IsValidVDSCommand, return: true, log id: f3b6397 >2012-11-08 06:43:04,855 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (pool-4-thread-49) START, IsVmDuringInitiatingVDSCommand( vmId = 67bd2d56-8e0b-4328-bcfd-ea7f4758c885), log id: e640b9c >2012-11-08 06:43:04,855 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (pool-4-thread-49) FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: e640b9c >2012-11-08 06:43:04,900 INFO [org.ovirt.engine.core.bll.RunVmOnceCommand] (pool-4-thread-49) Running command: RunVmOnceCommand internal: false. Entities affected : ID: 67bd2d56-8e0b-4328-bcfd-ea7f4758c885 Type: VM, ID: 67bd2d56-8e0b-4328-bcfd-ea7f4758c885 Type: VM >2012-11-08 06:43:04,968 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (pool-4-thread-49) START, CreateVmVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=67bd2d56-8e0b-4328-bcfd-ea7f4758c885, vm=org.ovirt.engine.core.common.businessentities.VM@e58b28b), log id: 69ef800b >2012-11-08 06:43:04,984 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (pool-4-thread-49) START, CreateVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=67bd2d56-8e0b-4328-bcfd-ea7f4758c885, vm=org.ovirt.engine.core.common.businessentities.VM@e58b28b), log id: 3001296a >2012-11-08 06:43:05,036 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (pool-4-thread-49) org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand spiceSslCipherSuite=DEFAULT,memSize=512,kvmEnable=true,smp=1,emulatedMachine=rhel6.3.0,vmType=kvm,keyboardLayout=en-us,nice=0,display=qxl,smpCoresPerSocket=1,spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard,timeOffset=-43200,transparentHugePages=true,vmId=67bd2d56-8e0b-4328-bcfd-ea7f4758c885,devices=[Ljava.util.Map;@49e0be48,acpiEnable=true,vmName=b1,cpuType=Nehalem,custom={viodiskcache=none} >2012-11-08 06:43:05,036 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (pool-4-thread-49) FINISH, CreateVDSCommand, log id: 3001296a >2012-11-08 06:43:05,036 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (pool-4-thread-49) IncreasePendingVms::CreateVmIncreasing vds rhs-gp-srv11.lab.eng.blr.redhat.com pending vcpu count, now 1. Vm: b1 >2012-11-08 06:43:05,038 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (pool-4-thread-49) FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 69ef800b >2012-11-08 06:43:06,727 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (QuartzScheduler_Worker-63) START, DestroyVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=67bd2d56-8e0b-4328-bcfd-ea7f4758c885, force=false, secondsToWait=0, gracefully=false), log id: 3f793bcc >2012-11-08 06:43:06,787 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (QuartzScheduler_Worker-63) FINISH, DestroyVDSCommand, log id: 3f793bcc >2012-11-08 06:43:06,817 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-63) Running on vds during rerun failed vm: null >2012-11-08 06:43:06,818 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-63) vm b1 running in db and not running in vds - add to rerun treatment. vds rhs-gp-srv11.lab.eng.blr.redhat.com >2012-11-08 06:43:06,824 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-63) Rerun vm 67bd2d56-8e0b-4328-bcfd-ea7f4758c885. Called from vds rhs-gp-srv11.lab.eng.blr.redhat.com >2012-11-08 06:43:06,847 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsDynamicDataVDSCommand] (pool-4-thread-48) START, UpdateVdsDynamicDataVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vdsDynamic=org.ovirt.engine.core.common.businessentities.VdsDynamic@a701a1d5), log id: 296540a8 >2012-11-08 06:43:06,849 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsDynamicDataVDSCommand] (pool-4-thread-48) FINISH, UpdateVdsDynamicDataVDSCommand, log id: 296540a8 >2012-11-08 06:43:06,877 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (pool-4-thread-48) START, IsValidVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 49f373c6 >2012-11-08 06:43:06,879 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (pool-4-thread-48) FINISH, IsValidVDSCommand, return: true, log id: 49f373c6 >2012-11-08 06:43:06,888 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (pool-4-thread-48) START, IsVmDuringInitiatingVDSCommand( vmId = 67bd2d56-8e0b-4328-bcfd-ea7f4758c885), log id: 6000299 >2012-11-08 06:43:06,888 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (pool-4-thread-48) FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 6000299 >2012-11-08 06:43:06,924 INFO [org.ovirt.engine.core.bll.VdsSelector] (pool-4-thread-48) VDS rhs-gp-srv11.lab.eng.blr.redhat.com 96739be4-2990-11e2-8577-52540035ed48 have failed running this VM in the current selection cycle VDS rhs-gp-srv4.lab.eng.blr.redhat.com d2bd32d4-2992-11e2-ab59-52540035ed48 is the same host the VM is currently running on VDS rhs-client11.lab.eng.blr.redhat.com 90b581c2-2977-11e2-82cd-52540035ed48 is not in up status or belongs to the VM's cluster >2012-11-08 06:43:06,924 WARN [org.ovirt.engine.core.bll.RunVmOnceCommand] (pool-4-thread-48) CanDoAction of action RunVmOnce failed. Reasons:VAR__ACTION__RUN,VAR__TYPE__VM,$VmName b1,VAR__ACTION__RUN,VAR__TYPE__VM,$VmName b1,VAR__ACTION__RUN,VAR__TYPE__VM,ACTION_TYPE_FAILED_VDS_VM_CLUSTER >2012-11-08 06:43:12,939 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-71) Failed to decrypt Data must not be longer than 256 bytes >2012-11-08 06:43:12,939 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-74) Failed to decrypt Data must not be longer than 256 bytes >2012-11-08 06:43:22,399 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-4) [4959d5e8] START, IsValidVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 143b01bb >2012-11-08 06:43:22,400 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-4) [4959d5e8] FINISH, IsValidVDSCommand, return: true, log id: 143b01bb >2012-11-08 06:43:22,407 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp-/127.0.0.1:8702-4) [4959d5e8] START, IsVmDuringInitiatingVDSCommand( vmId = 67bd2d56-8e0b-4328-bcfd-ea7f4758c885), log id: 1d7b3923 >2012-11-08 06:43:22,407 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp-/127.0.0.1:8702-4) [4959d5e8] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 1d7b3923 >2012-11-08 06:43:22,448 INFO [org.ovirt.engine.core.bll.RunVmOnceCommand] (ajp-/127.0.0.1:8702-4) [4959d5e8] Running command: RunVmOnceCommand internal: false. Entities affected : ID: 67bd2d56-8e0b-4328-bcfd-ea7f4758c885 Type: VM, ID: 67bd2d56-8e0b-4328-bcfd-ea7f4758c885 Type: VM >2012-11-08 06:43:22,516 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-4) [4959d5e8] START, CreateVmVDSCommand(HostName = rhs-gp-srv4.lab.eng.blr.redhat.com, HostId = d2bd32d4-2992-11e2-ab59-52540035ed48, vmId=67bd2d56-8e0b-4328-bcfd-ea7f4758c885, vm=org.ovirt.engine.core.common.businessentities.VM@47cfc1ce), log id: 44fbf870 >2012-11-08 06:43:22,526 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8702-4) [4959d5e8] START, CreateVDSCommand(HostName = rhs-gp-srv4.lab.eng.blr.redhat.com, HostId = d2bd32d4-2992-11e2-ab59-52540035ed48, vmId=67bd2d56-8e0b-4328-bcfd-ea7f4758c885, vm=org.ovirt.engine.core.common.businessentities.VM@47cfc1ce), log id: 59badbe >2012-11-08 06:43:22,572 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8702-4) [4959d5e8] org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand spiceSslCipherSuite=DEFAULT,memSize=512,kvmEnable=true,smp=1,emulatedMachine=rhel6.3.0,vmType=kvm,keyboardLayout=en-us,nice=0,display=qxl,smpCoresPerSocket=1,spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard,timeOffset=-43200,transparentHugePages=true,vmId=67bd2d56-8e0b-4328-bcfd-ea7f4758c885,devices=[Ljava.util.Map;@4335941c,acpiEnable=true,vmName=b1,cpuType=Nehalem,custom={viodiskcache=none} >2012-11-08 06:43:22,572 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8702-4) [4959d5e8] FINISH, CreateVDSCommand, log id: 59badbe >2012-11-08 06:43:22,573 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-4) [4959d5e8] IncreasePendingVms::CreateVmIncreasing vds rhs-gp-srv4.lab.eng.blr.redhat.com pending vcpu count, now 1. Vm: b1 >2012-11-08 06:43:22,575 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-4) [4959d5e8] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 44fbf870 >2012-11-08 06:43:23,134 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-95) Failed to decrypt Data must not be longer than 256 bytes >2012-11-08 06:43:23,134 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-93) Failed to decrypt Data must not be longer than 256 bytes >2012-11-08 06:43:25,228 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (QuartzScheduler_Worker-97) START, DestroyVDSCommand(HostName = rhs-gp-srv4.lab.eng.blr.redhat.com, HostId = d2bd32d4-2992-11e2-ab59-52540035ed48, vmId=67bd2d56-8e0b-4328-bcfd-ea7f4758c885, force=false, secondsToWait=0, gracefully=false), log id: 5d142ab1 >2012-11-08 06:43:25,281 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (QuartzScheduler_Worker-97) FINISH, DestroyVDSCommand, log id: 5d142ab1 >2012-11-08 06:43:25,321 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-97) Running on vds during rerun failed vm: null >2012-11-08 06:43:25,322 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-97) vm b1 running in db and not running in vds - add to rerun treatment. vds rhs-gp-srv4.lab.eng.blr.redhat.com >2012-11-08 06:43:25,328 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-97) Rerun vm 67bd2d56-8e0b-4328-bcfd-ea7f4758c885. Called from vds rhs-gp-srv4.lab.eng.blr.redhat.com >2012-11-08 06:43:25,368 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsDynamicDataVDSCommand] (pool-4-thread-48) START, UpdateVdsDynamicDataVDSCommand(HostName = rhs-gp-srv4.lab.eng.blr.redhat.com, HostId = d2bd32d4-2992-11e2-ab59-52540035ed48, vdsDynamic=org.ovirt.engine.core.common.businessentities.VdsDynamic@725ab7ae), log id: 4ff3611f >2012-11-08 06:43:25,370 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsDynamicDataVDSCommand] (pool-4-thread-48) FINISH, UpdateVdsDynamicDataVDSCommand, log id: 4ff3611f >2012-11-08 06:43:25,427 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (pool-4-thread-48) START, IsValidVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 68175844 >2012-11-08 06:43:25,428 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (pool-4-thread-48) FINISH, IsValidVDSCommand, return: true, log id: 68175844 >2012-11-08 06:43:25,466 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (pool-4-thread-48) START, IsVmDuringInitiatingVDSCommand( vmId = 67bd2d56-8e0b-4328-bcfd-ea7f4758c885), log id: 3541b1ec >2012-11-08 06:43:25,466 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (pool-4-thread-48) FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 3541b1ec >2012-11-08 06:43:25,518 INFO [org.ovirt.engine.core.bll.RunVmOnceCommand] (pool-4-thread-48) Running command: RunVmOnceCommand internal: false. Entities affected : ID: 67bd2d56-8e0b-4328-bcfd-ea7f4758c885 Type: VM, ID: 67bd2d56-8e0b-4328-bcfd-ea7f4758c885 Type: VM >2012-11-08 06:43:25,587 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (pool-4-thread-48) START, CreateVmVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=67bd2d56-8e0b-4328-bcfd-ea7f4758c885, vm=org.ovirt.engine.core.common.businessentities.VM@369ff9cc), log id: 2aa38464 >2012-11-08 06:43:25,599 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (pool-4-thread-48) START, CreateVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=67bd2d56-8e0b-4328-bcfd-ea7f4758c885, vm=org.ovirt.engine.core.common.businessentities.VM@369ff9cc), log id: 747dd94e >2012-11-08 06:43:25,637 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (pool-4-thread-48) org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand spiceSslCipherSuite=DEFAULT,memSize=512,kvmEnable=true,smp=1,emulatedMachine=rhel6.3.0,vmType=kvm,keyboardLayout=en-us,nice=0,display=qxl,smpCoresPerSocket=1,spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard,timeOffset=-43200,transparentHugePages=true,vmId=67bd2d56-8e0b-4328-bcfd-ea7f4758c885,devices=[Ljava.util.Map;@33869224,acpiEnable=true,vmName=b1,cpuType=Nehalem,custom={viodiskcache=none} >2012-11-08 06:43:25,637 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (pool-4-thread-48) FINISH, CreateVDSCommand, log id: 747dd94e >2012-11-08 06:43:25,638 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (pool-4-thread-48) IncreasePendingVms::CreateVmIncreasing vds rhs-gp-srv11.lab.eng.blr.redhat.com pending vcpu count, now 1. Vm: b1 >2012-11-08 06:43:25,643 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (pool-4-thread-48) FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 2aa38464 >2012-11-08 06:43:27,260 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (QuartzScheduler_Worker-7) START, DestroyVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=67bd2d56-8e0b-4328-bcfd-ea7f4758c885, force=false, secondsToWait=0, gracefully=false), log id: 755c6c3e >2012-11-08 06:43:27,312 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (QuartzScheduler_Worker-7) FINISH, DestroyVDSCommand, log id: 755c6c3e >2012-11-08 06:43:27,334 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-7) Running on vds during rerun failed vm: null >2012-11-08 06:43:27,335 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-7) vm b1 running in db and not running in vds - add to rerun treatment. vds rhs-gp-srv11.lab.eng.blr.redhat.com >2012-11-08 06:43:27,338 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-7) Rerun vm 67bd2d56-8e0b-4328-bcfd-ea7f4758c885. Called from vds rhs-gp-srv11.lab.eng.blr.redhat.com >2012-11-08 06:43:27,358 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsDynamicDataVDSCommand] (pool-4-thread-49) START, UpdateVdsDynamicDataVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vdsDynamic=org.ovirt.engine.core.common.businessentities.VdsDynamic@a701a1d5), log id: 3c85499f >2012-11-08 06:43:27,360 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsDynamicDataVDSCommand] (pool-4-thread-49) FINISH, UpdateVdsDynamicDataVDSCommand, log id: 3c85499f >2012-11-08 06:43:27,386 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (pool-4-thread-49) START, IsValidVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 406164ed >2012-11-08 06:43:27,388 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (pool-4-thread-49) FINISH, IsValidVDSCommand, return: true, log id: 406164ed >2012-11-08 06:43:27,398 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (pool-4-thread-49) START, IsVmDuringInitiatingVDSCommand( vmId = 67bd2d56-8e0b-4328-bcfd-ea7f4758c885), log id: 71e01a10 >2012-11-08 06:43:27,398 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (pool-4-thread-49) FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 71e01a10 >2012-11-08 06:43:27,438 INFO [org.ovirt.engine.core.bll.VdsSelector] (pool-4-thread-49) VDS rhs-gp-srv4.lab.eng.blr.redhat.com d2bd32d4-2992-11e2-ab59-52540035ed48 is the same host the VM is currently running on VDS rhs-gp-srv11.lab.eng.blr.redhat.com 96739be4-2990-11e2-8577-52540035ed48 have failed running this VM in the current selection cycle VDS rhs-client11.lab.eng.blr.redhat.com 90b581c2-2977-11e2-82cd-52540035ed48 is not in up status or belongs to the VM's cluster >2012-11-08 06:43:27,438 WARN [org.ovirt.engine.core.bll.RunVmOnceCommand] (pool-4-thread-49) CanDoAction of action RunVmOnce failed. Reasons:VAR__ACTION__RUN,VAR__TYPE__VM,$VmName b1,VAR__ACTION__RUN,VAR__TYPE__VM,$VmName b1,VAR__ACTION__RUN,VAR__TYPE__VM,ACTION_TYPE_FAILED_VDS_VM_CLUSTER >2012-11-08 06:43:59,963 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-66) Failed to decrypt Data must not be longer than 256 bytes >2012-11-08 06:43:59,963 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-70) Failed to decrypt Data must not be longer than 256 bytes >2012-11-08 06:45:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-73) Autorecovering hosts is disabled, skipping >2012-11-08 06:45:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-73) Autorecovering storage domains is disabled, skipping >2012-11-08 06:45:19,724 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-17) Failed to decrypt Data must not be longer than 256 bytes >2012-11-08 06:45:19,724 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-12) Failed to decrypt Data must not be longer than 256 bytes >2012-11-08 06:47:54,926 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-92) Failed to decrypt Data must not be longer than 256 bytes >2012-11-08 06:47:54,926 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-93) Failed to decrypt Data must not be longer than 256 bytes >2012-11-08 06:47:59,045 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-5) Failed to decrypt Data must not be longer than 256 bytes >2012-11-08 06:47:59,058 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-3) Failed to decrypt Data must start with zero >2012-11-08 06:48:03,105 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-14) Failed to decrypt Data must not be longer than 256 bytes >2012-11-08 06:48:03,105 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-11) Failed to decrypt Data must not be longer than 256 bytes >2012-11-08 06:49:45,224 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-95) Failed to decrypt Data must not be longer than 256 bytes >2012-11-08 06:49:45,224 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-97) Failed to decrypt Data must not be longer than 256 bytes >2012-11-08 06:49:56,750 INFO [org.ovirt.engine.core.bll.LogoutUserCommand] (ajp-/127.0.0.1:8702-13) [7abb1201] Running command: LogoutUserCommand internal: false. >2012-11-08 06:49:57,452 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-23) Failed to decrypt Data must not be longer than 256 bytes >2012-11-08 06:50:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-9) Autorecovering hosts is disabled, skipping >2012-11-08 06:50:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-9) Autorecovering storage domains is disabled, skipping >2012-11-08 06:50:03,584 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-31) Failed to decrypt Data must not be longer than 256 bytes >2012-11-08 06:50:03,584 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-36) Failed to decrypt Data must not be longer than 256 bytes >2012-11-08 06:50:07,633 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-42) Failed to decrypt Data must not be longer than 256 bytes >2012-11-08 06:50:07,633 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-47) Failed to decrypt Data must not be longer than 256 bytes >2012-11-08 06:51:13,019 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-65) Failed to decrypt Data must not be longer than 256 bytes >2012-11-08 06:51:13,019 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-64) Failed to decrypt Data must not be longer than 256 bytes >2012-11-08 06:52:06,136 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-56) Failed to decrypt Data must not be longer than 256 bytes >2012-11-08 06:52:06,136 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-58) Failed to decrypt Data must not be longer than 256 bytes >2012-11-08 06:52:08,169 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-64) Failed to decrypt Data must not be longer than 256 bytes >2012-11-08 06:52:08,169 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-65) Failed to decrypt Data must not be longer than 256 bytes >2012-11-08 06:54:33,181 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-28) Failed to decrypt Data must not be longer than 256 bytes >2012-11-08 06:54:33,181 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-25) Failed to decrypt Data must not be longer than 256 bytes >2012-11-08 06:55:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-80) Autorecovering hosts is disabled, skipping >2012-11-08 06:55:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-80) Autorecovering storage domains is disabled, skipping >2012-11-08 06:57:41,791 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-5) [11dee988] START, IsValidVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: e7e4027 >2012-11-08 06:57:41,791 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-5) [11dee988] FINISH, IsValidVDSCommand, return: true, log id: e7e4027 >2012-11-08 06:57:41,798 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp-/127.0.0.1:8702-5) [11dee988] START, IsVmDuringInitiatingVDSCommand( vmId = 67bd2d56-8e0b-4328-bcfd-ea7f4758c885), log id: 1c81d99e >2012-11-08 06:57:41,798 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp-/127.0.0.1:8702-5) [11dee988] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 1c81d99e >2012-11-08 06:57:41,840 INFO [org.ovirt.engine.core.bll.RunVmOnceCommand] (ajp-/127.0.0.1:8702-5) [11dee988] Running command: RunVmOnceCommand internal: false. Entities affected : ID: 67bd2d56-8e0b-4328-bcfd-ea7f4758c885 Type: VM, ID: 67bd2d56-8e0b-4328-bcfd-ea7f4758c885 Type: VM >2012-11-08 06:57:41,913 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-5) [11dee988] START, CreateVmVDSCommand(HostName = rhs-gp-srv4.lab.eng.blr.redhat.com, HostId = d2bd32d4-2992-11e2-ab59-52540035ed48, vmId=67bd2d56-8e0b-4328-bcfd-ea7f4758c885, vm=org.ovirt.engine.core.common.businessentities.VM@5103e094), log id: 13362c92 >2012-11-08 06:57:41,927 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8702-5) [11dee988] START, CreateVDSCommand(HostName = rhs-gp-srv4.lab.eng.blr.redhat.com, HostId = d2bd32d4-2992-11e2-ab59-52540035ed48, vmId=67bd2d56-8e0b-4328-bcfd-ea7f4758c885, vm=org.ovirt.engine.core.common.businessentities.VM@5103e094), log id: 643646fe >2012-11-08 06:57:41,970 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8702-5) [11dee988] org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand spiceSslCipherSuite=DEFAULT,memSize=512,kvmEnable=true,smp=1,emulatedMachine=rhel6.3.0,vmType=kvm,keyboardLayout=en-us,nice=0,display=qxl,smpCoresPerSocket=1,spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard,timeOffset=-43200,transparentHugePages=true,vmId=67bd2d56-8e0b-4328-bcfd-ea7f4758c885,devices=[Ljava.util.Map;@4e0aefd6,acpiEnable=true,vmName=b1,cpuType=Nehalem,custom={viodiskcache=none} >2012-11-08 06:57:41,970 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8702-5) [11dee988] FINISH, CreateVDSCommand, log id: 643646fe >2012-11-08 06:57:41,970 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-5) [11dee988] IncreasePendingVms::CreateVmIncreasing vds rhs-gp-srv4.lab.eng.blr.redhat.com pending vcpu count, now 1. Vm: b1 >2012-11-08 06:57:41,972 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-5) [11dee988] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 13362c92 >2012-11-08 06:57:43,543 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (QuartzScheduler_Worker-80) START, DestroyVDSCommand(HostName = rhs-gp-srv4.lab.eng.blr.redhat.com, HostId = d2bd32d4-2992-11e2-ab59-52540035ed48, vmId=67bd2d56-8e0b-4328-bcfd-ea7f4758c885, force=false, secondsToWait=0, gracefully=false), log id: 7a9e5a52 >2012-11-08 06:57:43,595 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (QuartzScheduler_Worker-80) FINISH, DestroyVDSCommand, log id: 7a9e5a52 >2012-11-08 06:57:43,619 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-80) Running on vds during rerun failed vm: null >2012-11-08 06:57:43,620 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-80) vm b1 running in db and not running in vds - add to rerun treatment. vds rhs-gp-srv4.lab.eng.blr.redhat.com >2012-11-08 06:57:43,623 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-80) Rerun vm 67bd2d56-8e0b-4328-bcfd-ea7f4758c885. Called from vds rhs-gp-srv4.lab.eng.blr.redhat.com >2012-11-08 06:57:43,649 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsDynamicDataVDSCommand] (pool-4-thread-49) START, UpdateVdsDynamicDataVDSCommand(HostName = rhs-gp-srv4.lab.eng.blr.redhat.com, HostId = d2bd32d4-2992-11e2-ab59-52540035ed48, vdsDynamic=org.ovirt.engine.core.common.businessentities.VdsDynamic@34fff0ec), log id: 16b2b308 >2012-11-08 06:57:43,651 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsDynamicDataVDSCommand] (pool-4-thread-49) FINISH, UpdateVdsDynamicDataVDSCommand, log id: 16b2b308 >2012-11-08 06:57:43,669 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (pool-4-thread-49) START, IsValidVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 4089591e >2012-11-08 06:57:43,670 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (pool-4-thread-49) FINISH, IsValidVDSCommand, return: true, log id: 4089591e >2012-11-08 06:57:43,676 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (pool-4-thread-49) START, IsVmDuringInitiatingVDSCommand( vmId = 67bd2d56-8e0b-4328-bcfd-ea7f4758c885), log id: 543ea905 >2012-11-08 06:57:43,676 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (pool-4-thread-49) FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 543ea905 >2012-11-08 06:57:43,714 INFO [org.ovirt.engine.core.bll.RunVmOnceCommand] (pool-4-thread-49) Running command: RunVmOnceCommand internal: false. Entities affected : ID: 67bd2d56-8e0b-4328-bcfd-ea7f4758c885 Type: VM, ID: 67bd2d56-8e0b-4328-bcfd-ea7f4758c885 Type: VM >2012-11-08 06:57:43,788 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (pool-4-thread-49) START, CreateVmVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=67bd2d56-8e0b-4328-bcfd-ea7f4758c885, vm=org.ovirt.engine.core.common.businessentities.VM@6714f5c2), log id: 552b41a6 >2012-11-08 06:57:43,800 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (pool-4-thread-49) START, CreateVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=67bd2d56-8e0b-4328-bcfd-ea7f4758c885, vm=org.ovirt.engine.core.common.businessentities.VM@6714f5c2), log id: 7a5952ea >2012-11-08 06:57:43,846 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (pool-4-thread-49) org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand spiceSslCipherSuite=DEFAULT,memSize=512,kvmEnable=true,smp=1,emulatedMachine=rhel6.3.0,vmType=kvm,keyboardLayout=en-us,nice=0,display=qxl,smpCoresPerSocket=1,spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard,timeOffset=-43200,transparentHugePages=true,vmId=67bd2d56-8e0b-4328-bcfd-ea7f4758c885,devices=[Ljava.util.Map;@36246d28,acpiEnable=true,vmName=b1,cpuType=Nehalem,custom={viodiskcache=none} >2012-11-08 06:57:43,846 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (pool-4-thread-49) FINISH, CreateVDSCommand, log id: 7a5952ea >2012-11-08 06:57:43,847 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (pool-4-thread-49) IncreasePendingVms::CreateVmIncreasing vds rhs-gp-srv11.lab.eng.blr.redhat.com pending vcpu count, now 1. Vm: b1 >2012-11-08 06:57:43,850 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (pool-4-thread-49) FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 552b41a6 >2012-11-08 06:57:45,394 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (QuartzScheduler_Worker-81) START, DestroyVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=67bd2d56-8e0b-4328-bcfd-ea7f4758c885, force=false, secondsToWait=0, gracefully=false), log id: 630244f5 >2012-11-08 06:57:45,450 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (QuartzScheduler_Worker-81) FINISH, DestroyVDSCommand, log id: 630244f5 >2012-11-08 06:57:45,472 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-81) Running on vds during rerun failed vm: null >2012-11-08 06:57:45,473 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-81) vm b1 running in db and not running in vds - add to rerun treatment. vds rhs-gp-srv11.lab.eng.blr.redhat.com >2012-11-08 06:57:45,476 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-81) Rerun vm 67bd2d56-8e0b-4328-bcfd-ea7f4758c885. Called from vds rhs-gp-srv11.lab.eng.blr.redhat.com >2012-11-08 06:57:45,496 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsDynamicDataVDSCommand] (pool-4-thread-47) START, UpdateVdsDynamicDataVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vdsDynamic=org.ovirt.engine.core.common.businessentities.VdsDynamic@a701a1d5), log id: 40b52717 >2012-11-08 06:57:45,498 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsDynamicDataVDSCommand] (pool-4-thread-47) FINISH, UpdateVdsDynamicDataVDSCommand, log id: 40b52717 >2012-11-08 06:57:45,530 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (pool-4-thread-47) START, IsValidVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 2a42b47d >2012-11-08 06:57:45,531 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (pool-4-thread-47) FINISH, IsValidVDSCommand, return: true, log id: 2a42b47d >2012-11-08 06:57:45,541 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (pool-4-thread-47) START, IsVmDuringInitiatingVDSCommand( vmId = 67bd2d56-8e0b-4328-bcfd-ea7f4758c885), log id: 23766848 >2012-11-08 06:57:45,541 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (pool-4-thread-47) FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 23766848 >2012-11-08 06:57:45,580 INFO [org.ovirt.engine.core.bll.VdsSelector] (pool-4-thread-47) VDS rhs-gp-srv11.lab.eng.blr.redhat.com 96739be4-2990-11e2-8577-52540035ed48 have failed running this VM in the current selection cycle VDS rhs-client11.lab.eng.blr.redhat.com 90b581c2-2977-11e2-82cd-52540035ed48 is not in up status or belongs to the VM's cluster VDS rhs-gp-srv4.lab.eng.blr.redhat.com d2bd32d4-2992-11e2-ab59-52540035ed48 is the same host the VM is currently running on >2012-11-08 06:57:45,580 WARN [org.ovirt.engine.core.bll.RunVmOnceCommand] (pool-4-thread-47) CanDoAction of action RunVmOnce failed. Reasons:VAR__ACTION__RUN,VAR__TYPE__VM,$VmName b1,VAR__ACTION__RUN,VAR__TYPE__VM,$VmName b1,VAR__ACTION__RUN,VAR__TYPE__VM,ACTION_TYPE_FAILED_VDS_VM_CLUSTER >2012-11-08 06:58:10,073 INFO [org.ovirt.engine.core.bll.LoginAdminUserCommand] (ajp-/127.0.0.1:8702-1) Checking if user admin@internal is an admin, result true >2012-11-08 06:58:10,073 INFO [org.ovirt.engine.core.bll.LoginAdminUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginAdminUserCommand internal: false. >2012-11-08 06:58:21,263 INFO [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (ajp-/127.0.0.1:8702-11) [e8985e2] Lock Acquired to object EngineLock [exclusiveLocks= key: b2 value: VM_NAME >, sharedLocks= ] >2012-11-08 06:58:21,276 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-11) [e8985e2] START, IsValidVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 1a6d566 >2012-11-08 06:58:21,277 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-11) [e8985e2] FINISH, IsValidVDSCommand, return: true, log id: 1a6d566 >2012-11-08 06:58:21,281 INFO [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (ajp-/127.0.0.1:8702-11) [e8985e2] Running command: AddVmFromScratchCommand internal: false. Entities affected : ID: 6cea47a6-2976-11e2-ad99-52540035ed48 Type: VdsGroups, ID: 6cea47a6-2976-11e2-ad99-52540035ed48 Type: VdsGroups >2012-11-08 06:58:21,282 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-11) [e8985e2] START, IsValidVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 47fbac3b >2012-11-08 06:58:21,282 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-11) [e8985e2] FINISH, IsValidVDSCommand, return: true, log id: 47fbac3b >2012-11-08 06:58:21,298 INFO [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (ajp-/127.0.0.1:8702-11) [e8985e2] Lock freed to object EngineLock [exclusiveLocks= key: b2 value: VM_NAME >, sharedLocks= ] >2012-11-08 06:58:30,716 INFO [org.ovirt.engine.core.bll.AddDiskCommand] (ajp-/127.0.0.1:8702-12) [4d3f2020] Lock Acquired to object EngineLock [exclusiveLocks= key: 68655de5-7bd9-4bdc-8628-3b2069ec312f value: VM_DISK_BOOT >, sharedLocks= key: 68655de5-7bd9-4bdc-8628-3b2069ec312f value: VM >] >2012-11-08 06:58:30,719 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-12) [4d3f2020] START, IsValidVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 4bdba512 >2012-11-08 06:58:30,720 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-12) [4d3f2020] FINISH, IsValidVDSCommand, return: true, log id: 4bdba512 >2012-11-08 06:58:30,725 INFO [org.ovirt.engine.core.bll.AddDiskCommand] (ajp-/127.0.0.1:8702-12) [4d3f2020] Running command: AddDiskCommand internal: false. Entities affected : ID: 68655de5-7bd9-4bdc-8628-3b2069ec312f Type: VM, ID: a02e96be-cf4c-4d0f-8368-fd9d4aa659bf Type: Storage >2012-11-08 06:58:30,742 INFO [org.ovirt.engine.core.bll.AddImageFromScratchCommand] (ajp-/127.0.0.1:8702-12) [326743ea] Running command: AddImageFromScratchCommand internal: true. Entities affected : ID: a02e96be-cf4c-4d0f-8368-fd9d4aa659bf Type: Storage >2012-11-08 06:58:30,753 INFO [org.ovirt.engine.core.bll.AddImageFromScratchCommand] (ajp-/127.0.0.1:8702-12) [326743ea] Lock freed to object EngineLock [exclusiveLocks= key: 68655de5-7bd9-4bdc-8628-3b2069ec312f value: VM_DISK_BOOT >, sharedLocks= key: 68655de5-7bd9-4bdc-8628-3b2069ec312f value: VM >] >2012-11-08 06:58:30,755 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (ajp-/127.0.0.1:8702-12) [326743ea] START, CreateImageVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = 3.1, storageDomainId = a02e96be-cf4c-4d0f-8368-fd9d4aa659bf, imageGroupId = 8225325a-7c64-410e-b68a-7f762ce8d2e3, imageSizeInBytes = 5368709120, volumeFormat = RAW, newImageId = 78fb3c92-5992-4d05-8b72-3c699a0b0956, newImageDescription = ), log id: 3fc4d17f >2012-11-08 06:58:30,755 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (ajp-/127.0.0.1:8702-12) [326743ea] -- CreateImageVDSCommand::ExecuteIrsBrokerCommand: calling 'createVolume' with two new parameters: description and UUID >2012-11-08 06:58:30,755 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (ajp-/127.0.0.1:8702-12) [326743ea] -- createVolume parameters: > sdUUID=a02e96be-cf4c-4d0f-8368-fd9d4aa659bf > spUUID=3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 > imgGUID=8225325a-7c64-410e-b68a-7f762ce8d2e3 > size=5,368,709,120 bytes > volFormat=RAW > volType=Preallocated > volUUID=78fb3c92-5992-4d05-8b72-3c699a0b0956 > descr= > srcImgGUID=00000000-0000-0000-0000-000000000000 > srcVolUUID=00000000-0000-0000-0000-000000000000 > >2012-11-08 06:58:30,838 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (ajp-/127.0.0.1:8702-12) [326743ea] FINISH, CreateImageVDSCommand, return: 78fb3c92-5992-4d05-8b72-3c699a0b0956, log id: 3fc4d17f >2012-11-08 06:58:30,841 INFO [org.ovirt.engine.core.bll.EntityAsyncTask] (ajp-/127.0.0.1:8702-12) [326743ea] EntityAsyncTask::Adding EntityMultiAsyncTasks object for entity 8225325a-7c64-410e-b68a-7f762ce8d2e3 >2012-11-08 06:58:30,841 INFO [org.ovirt.engine.core.bll.EntityMultiAsyncTasks] (ajp-/127.0.0.1:8702-12) [326743ea] EntityMultiAsyncTasks::AttachTask: Attaching task 65819c56-68d2-474a-964c-d10de8f32803 to entity 8225325a-7c64-410e-b68a-7f762ce8d2e3. >2012-11-08 06:58:30,842 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskUtils] (ajp-/127.0.0.1:8702-12) [326743ea] Adding task 65819c56-68d2-474a-964c-d10de8f32803 to DataBase >2012-11-08 06:58:30,886 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (ajp-/127.0.0.1:8702-12) [326743ea] Adding task 65819c56-68d2-474a-964c-d10de8f32803 (Parent Command AddDisk, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters), polling hasn't started yet.. >2012-11-08 06:58:30,893 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (ajp-/127.0.0.1:8702-12) [326743ea] BaseAsyncTask::StartPollingTask: Starting to poll task 65819c56-68d2-474a-964c-d10de8f32803. >2012-11-08 06:58:30,893 INFO [org.ovirt.engine.core.bll.EntityMultiAsyncTasks] (ajp-/127.0.0.1:8702-12) [326743ea] EntityMultiAsyncTasks::StartPollingTask: Current Action Type for entity 8225325a-7c64-410e-b68a-7f762ce8d2e3 is AddDisk (determined by task 65819c56-68d2-474a-964c-d10de8f32803) >2012-11-08 06:58:36,844 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-80) Polling and updating Async Tasks: 1 tasks, 1 tasks to poll now >2012-11-08 06:58:36,876 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (QuartzScheduler_Worker-80) SPMAsyncTask::PollTask: Polling task 65819c56-68d2-474a-964c-d10de8f32803 (Parent Command AddDisk, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status running. >2012-11-08 06:58:36,876 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-80) Finished polling Tasks, will poll again in 10 seconds. >2012-11-08 06:58:40,907 INFO [org.ovirt.engine.core.bll.AddVmInterfaceCommand] (ajp-/127.0.0.1:8702-12) [534a5af5] Running command: AddVmInterfaceCommand internal: false. Entities affected : ID: 68655de5-7bd9-4bdc-8628-3b2069ec312f Type: VM >2012-11-08 06:58:40,908 INFO [org.ovirt.engine.core.bll.MacPoolManager] (ajp-/127.0.0.1:8702-12) [534a5af5] MacPoolManager::allocateNewMac entered >2012-11-08 06:58:40,908 INFO [org.ovirt.engine.core.bll.MacPoolManager] (ajp-/127.0.0.1:8702-12) [534a5af5] MacPoolManager::allocateNewMac allocated mac = 00:1a:4a:46:22:4e >2012-11-08 06:58:40,932 INFO [org.ovirt.engine.core.bll.ActivateDeactivateVmNicCommand] (ajp-/127.0.0.1:8702-12) [5148f4a0] Running command: ActivateDeactivateVmNicCommand internal: true. Entities affected : ID: 68655de5-7bd9-4bdc-8628-3b2069ec312f Type: VM >2012-11-08 06:59:27,059 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (QuartzScheduler_Worker-70) SPMAsyncTask::PollTask: Polling task 65819c56-68d2-474a-964c-d10de8f32803 (Parent Command AddDisk, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status finished, result 'success'. >2012-11-08 06:59:27,063 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (QuartzScheduler_Worker-70) BaseAsyncTask::OnTaskEndSuccess: Task 65819c56-68d2-474a-964c-d10de8f32803 (Parent Command AddDisk, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended successfully. >2012-11-08 06:59:27,064 INFO [org.ovirt.engine.core.bll.EntityAsyncTask] (QuartzScheduler_Worker-70) EntityAsyncTask::EndActionIfNecessary: All tasks of entity 8225325a-7c64-410e-b68a-7f762ce8d2e3 has ended -> executing EndAction >2012-11-08 06:59:27,064 INFO [org.ovirt.engine.core.bll.EntityAsyncTask] (QuartzScheduler_Worker-70) EntityAsyncTask::EndAction: Ending action for 1 tasks (entity ID: 8225325a-7c64-410e-b68a-7f762ce8d2e3): calling EndAction for action type AddDisk. >2012-11-08 06:59:27,064 INFO [org.ovirt.engine.core.bll.EntityAsyncTask] (pool-4-thread-47) EntityAsyncTask::EndCommandAction [within thread]context: Attempting to EndAction AddDisk >2012-11-08 06:59:27,080 INFO [org.ovirt.engine.core.bll.AddDiskCommand] (pool-4-thread-47) [4d3f2020] Ending command successfully: org.ovirt.engine.core.bll.AddDiskCommand >2012-11-08 06:59:27,080 INFO [org.ovirt.engine.core.bll.AddImageFromScratchCommand] (pool-4-thread-47) [326743ea] Ending command successfully: org.ovirt.engine.core.bll.AddImageFromScratchCommand >2012-11-08 06:59:27,085 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (pool-4-thread-47) [326743ea] START, GetImageInfoVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null, storageDomainId = a02e96be-cf4c-4d0f-8368-fd9d4aa659bf, imageGroupId = 8225325a-7c64-410e-b68a-7f762ce8d2e3, imageId = 78fb3c92-5992-4d05-8b72-3c699a0b0956), log id: 6a53c01f >2012-11-08 06:59:27,131 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (pool-4-thread-47) [326743ea] FINISH, GetImageInfoVDSCommand, return: org.ovirt.engine.core.common.businessentities.DiskImage@f5afcbe6, log id: 6a53c01f >2012-11-08 06:59:27,148 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UpdateVMVDSCommand] (pool-4-thread-47) START, UpdateVMVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null, storageDomainId = 00000000-0000-0000-0000-000000000000, infoDictionary.size = 1), log id: 683534d5 >2012-11-08 06:59:27,178 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UpdateVMVDSCommand] (pool-4-thread-47) FINISH, UpdateVMVDSCommand, log id: 683534d5 >2012-11-08 06:59:27,186 INFO [org.ovirt.engine.core.bll.EntityAsyncTask] (pool-4-thread-47) EntityAsyncTask::HandleEndActionResult [within thread]: EndAction for action type AddDisk completed, handling the result. >2012-11-08 06:59:27,186 INFO [org.ovirt.engine.core.bll.EntityAsyncTask] (pool-4-thread-47) EntityAsyncTask::HandleEndActionResult [within thread]: EndAction for action type AddDisk succeeded, clearing tasks. >2012-11-08 06:59:27,188 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (pool-4-thread-47) SPMAsyncTask::ClearAsyncTask: Attempting to clear task 65819c56-68d2-474a-964c-d10de8f32803 >2012-11-08 06:59:27,188 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (pool-4-thread-47) START, SPMClearTaskVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null, taskId = 65819c56-68d2-474a-964c-d10de8f32803), log id: 365ac7d1 >2012-11-08 06:59:27,198 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (pool-4-thread-47) START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=65819c56-68d2-474a-964c-d10de8f32803), log id: 5ee635f6 >2012-11-08 06:59:27,225 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (pool-4-thread-47) FINISH, HSMClearTaskVDSCommand, log id: 5ee635f6 >2012-11-08 06:59:27,225 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (pool-4-thread-47) FINISH, SPMClearTaskVDSCommand, log id: 365ac7d1 >2012-11-08 06:59:27,227 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (pool-4-thread-47) BaseAsyncTask::RemoveTaskFromDB: Removed task 65819c56-68d2-474a-964c-d10de8f32803 from DataBase >2012-11-08 06:59:27,227 INFO [org.ovirt.engine.core.bll.EntityAsyncTask] (pool-4-thread-47) EntityAsyncTask::HandleEndActionResult [within thread]: Removing EntityMultiAsyncTasks object for entity 8225325a-7c64-410e-b68a-7f762ce8d2e3 >2012-11-08 07:00:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-32) Autorecovering hosts is disabled, skipping >2012-11-08 07:00:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-32) Autorecovering storage domains is disabled, skipping >2012-11-08 07:00:00,474 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-9) Failed to decrypt Data must not be longer than 256 bytes >2012-11-08 07:00:00,474 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-33) Failed to decrypt Data must not be longer than 256 bytes >2012-11-08 07:00:52,964 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (ajp-/127.0.0.1:8702-2) [5b27465] START, ValidateStorageServerConnectionVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = POSIXFS, connectionList = [{ id: null, connection: rhs-client37.lab.eng.blr.redhat.com:dist-rep, iqn: null, vfsType: glusterfs, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: b228001 >2012-11-08 07:00:52,984 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (ajp-/127.0.0.1:8702-2) [5b27465] FINISH, ValidateStorageServerConnectionVDSCommand, return: {00000000-0000-0000-0000-000000000000=0}, log id: b228001 >2012-11-08 07:00:52,987 INFO [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-2) [5b27465] Running command: AddStorageServerConnectionCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2012-11-08 07:00:53,003 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-2) [5b27465] START, ConnectStorageServerVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = POSIXFS, connectionList = [{ id: null, connection: rhs-client37.lab.eng.blr.redhat.com:dist-rep, iqn: null, vfsType: glusterfs, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 3054e9a4 >2012-11-08 07:00:56,253 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-33) Setting new tasks map. The map contains now 0 tasks >2012-11-08 07:00:56,253 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-33) Cleared all tasks of pool 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6. >2012-11-08 07:00:56,270 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-33) Could not find vds that is spm and non-operational. >2012-11-08 07:00:57,100 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-2) [5b27465] FINISH, ConnectStorageServerVDSCommand, return: {00000000-0000-0000-0000-000000000000=0}, log id: 3054e9a4 >2012-11-08 07:00:57,670 WARN [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (ajp-/127.0.0.1:8702-9) [699370b9] The message key AddPosixFsStorageDomain is missing from bundles/ExecutionMessages >2012-11-08 07:00:57,682 INFO [org.ovirt.engine.core.bll.storage.AddPosixFsStorageDomainCommand] (ajp-/127.0.0.1:8702-9) [699370b9] Running command: AddPosixFsStorageDomainCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2012-11-08 07:00:57,697 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-9) [699370b9] START, CreateStorageDomainVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storageDomain=org.ovirt.engine.core.common.businessentities.storage_domain_static@fed0339e, args=rhs-client37.lab.eng.blr.redhat.com:dist-rep), log id: 287cbb81 >2012-11-08 07:01:00,585 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-9) [699370b9] FINISH, CreateStorageDomainVDSCommand, log id: 287cbb81 >2012-11-08 07:01:00,596 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStorageDomainStatsVDSCommand] (ajp-/127.0.0.1:8702-9) [699370b9] START, GetStorageDomainStatsVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storageDomainId=e0cce407-ebea-4322-b47e-7aceb73a7a6f), log id: 5691be56 >2012-11-08 07:01:00,618 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStorageDomainStatsVDSCommand] (ajp-/127.0.0.1:8702-9) [699370b9] FINISH, GetStorageDomainStatsVDSCommand, return: org.ovirt.engine.core.common.businessentities.storage_domains@edd36781, log id: 5691be56 >2012-11-08 07:01:01,715 INFO [org.ovirt.engine.core.bll.storage.AttachStorageDomainToPoolCommand] (ajp-/127.0.0.1:8702-1) [3c99ec2] Running command: AttachStorageDomainToPoolCommand internal: false. Entities affected : ID: e0cce407-ebea-4322-b47e-7aceb73a7a6f Type: Storage >2012-11-08 07:01:01,791 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (pool-4-thread-48) [6a5bcba2] START, ValidateStorageServerConnectionVDSCommand(HostName = rhs-gp-srv4.lab.eng.blr.redhat.com, HostId = d2bd32d4-2992-11e2-ab59-52540035ed48, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = POSIXFS, connectionList = [{ id: 0d9a1047-add1-41b9-911c-bf284b6fbfb2, connection: rhs-client37.lab.eng.blr.redhat.com:dist-rep, iqn: null, vfsType: glusterfs, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 478b3a87 >2012-11-08 07:01:01,811 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (pool-4-thread-49) [521f02b6] START, ValidateStorageServerConnectionVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = POSIXFS, connectionList = [{ id: 0d9a1047-add1-41b9-911c-bf284b6fbfb2, connection: rhs-client37.lab.eng.blr.redhat.com:dist-rep, iqn: null, vfsType: glusterfs, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 5bd93658 >2012-11-08 07:01:01,814 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (pool-4-thread-48) [6a5bcba2] FINISH, ValidateStorageServerConnectionVDSCommand, return: {0d9a1047-add1-41b9-911c-bf284b6fbfb2=0}, log id: 478b3a87 >2012-11-08 07:01:01,814 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (pool-4-thread-48) [6a5bcba2] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2012-11-08 07:01:01,830 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-4-thread-48) [6a5bcba2] START, ConnectStorageServerVDSCommand(HostName = rhs-gp-srv4.lab.eng.blr.redhat.com, HostId = d2bd32d4-2992-11e2-ab59-52540035ed48, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = POSIXFS, connectionList = [{ id: 0d9a1047-add1-41b9-911c-bf284b6fbfb2, connection: rhs-client37.lab.eng.blr.redhat.com:dist-rep, iqn: null, vfsType: glusterfs, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 52b2ed8d >2012-11-08 07:01:01,847 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (pool-4-thread-49) [521f02b6] FINISH, ValidateStorageServerConnectionVDSCommand, return: {0d9a1047-add1-41b9-911c-bf284b6fbfb2=0}, log id: 5bd93658 >2012-11-08 07:01:01,847 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (pool-4-thread-49) [521f02b6] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2012-11-08 07:01:01,860 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-4-thread-49) [521f02b6] START, ConnectStorageServerVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = POSIXFS, connectionList = [{ id: 0d9a1047-add1-41b9-911c-bf284b6fbfb2, connection: rhs-client37.lab.eng.blr.redhat.com:dist-rep, iqn: null, vfsType: glusterfs, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 715f6c93 >2012-11-08 07:01:01,884 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-4-thread-49) [521f02b6] FINISH, ConnectStorageServerVDSCommand, return: {0d9a1047-add1-41b9-911c-bf284b6fbfb2=0}, log id: 715f6c93 >2012-11-08 07:01:05,922 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-4-thread-48) [6a5bcba2] FINISH, ConnectStorageServerVDSCommand, return: {0d9a1047-add1-41b9-911c-bf284b6fbfb2=0}, log id: 52b2ed8d >2012-11-08 07:01:05,924 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.AttachStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-1) [3c99ec2] START, AttachStorageDomainVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null, storageDomainId = e0cce407-ebea-4322-b47e-7aceb73a7a6f), log id: 715a6e43 >2012-11-08 07:01:30,101 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.AttachStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-1) [3c99ec2] FINISH, AttachStorageDomainVDSCommand, log id: 715a6e43 >2012-11-08 07:01:42,633 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-16) Failed to decrypt Data must not be longer than 256 bytes >2012-11-08 07:01:42,633 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-18) Failed to decrypt Data must not be longer than 256 bytes >2012-11-08 07:01:52,835 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-31) Failed to decrypt Data must not be longer than 256 bytes >2012-11-08 07:01:52,835 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-36) Failed to decrypt Data must not be longer than 256 bytes >2012-11-08 07:02:03,031 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-57) Failed to decrypt Data must not be longer than 256 bytes >2012-11-08 07:02:03,031 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-55) Failed to decrypt Data must not be longer than 256 bytes >2012-11-08 07:03:54,635 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp-/127.0.0.1:8702-7) Lock Acquired to object EngineLock [exclusiveLocks= key: e0cce407-ebea-4322-b47e-7aceb73a7a6f value: STORAGE >, sharedLocks= ] >2012-11-08 07:03:54,656 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (pool-4-thread-48) [72e2c6f5] Running command: ActivateStorageDomainCommand internal: false. Entities affected : ID: e0cce407-ebea-4322-b47e-7aceb73a7a6f Type: Storage >2012-11-08 07:03:54,666 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (pool-4-thread-48) [72e2c6f5] Lock freed to object EngineLock [exclusiveLocks= key: e0cce407-ebea-4322-b47e-7aceb73a7a6f value: STORAGE >, sharedLocks= ] >2012-11-08 07:03:54,666 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (pool-4-thread-48) [72e2c6f5] ActivateStorage Domain. Before Connect all hosts to pool. Time:11/8/12 7:03 AM >2012-11-08 07:03:54,721 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (pool-4-thread-49) [3f59adf9] START, ValidateStorageServerConnectionVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = POSIXFS, connectionList = [{ id: 0d9a1047-add1-41b9-911c-bf284b6fbfb2, connection: rhs-client37.lab.eng.blr.redhat.com:dist-rep, iqn: null, vfsType: glusterfs, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 3c404ca6 >2012-11-08 07:03:54,722 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (pool-4-thread-50) [113ca761] START, ValidateStorageServerConnectionVDSCommand(HostName = rhs-gp-srv4.lab.eng.blr.redhat.com, HostId = d2bd32d4-2992-11e2-ab59-52540035ed48, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = POSIXFS, connectionList = [{ id: 0d9a1047-add1-41b9-911c-bf284b6fbfb2, connection: rhs-client37.lab.eng.blr.redhat.com:dist-rep, iqn: null, vfsType: glusterfs, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 4f24b69d >2012-11-08 07:03:54,749 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (pool-4-thread-50) [113ca761] FINISH, ValidateStorageServerConnectionVDSCommand, return: {0d9a1047-add1-41b9-911c-bf284b6fbfb2=0}, log id: 4f24b69d >2012-11-08 07:03:54,749 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (pool-4-thread-49) [3f59adf9] FINISH, ValidateStorageServerConnectionVDSCommand, return: {0d9a1047-add1-41b9-911c-bf284b6fbfb2=0}, log id: 3c404ca6 >2012-11-08 07:03:54,749 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (pool-4-thread-50) [113ca761] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2012-11-08 07:03:54,750 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (pool-4-thread-49) [3f59adf9] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2012-11-08 07:03:54,764 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-4-thread-49) [3f59adf9] START, ConnectStorageServerVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = POSIXFS, connectionList = [{ id: 0d9a1047-add1-41b9-911c-bf284b6fbfb2, connection: rhs-client37.lab.eng.blr.redhat.com:dist-rep, iqn: null, vfsType: glusterfs, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 2272ecb2 >2012-11-08 07:03:54,769 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-4-thread-50) [113ca761] START, ConnectStorageServerVDSCommand(HostName = rhs-gp-srv4.lab.eng.blr.redhat.com, HostId = d2bd32d4-2992-11e2-ab59-52540035ed48, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = POSIXFS, connectionList = [{ id: 0d9a1047-add1-41b9-911c-bf284b6fbfb2, connection: rhs-client37.lab.eng.blr.redhat.com:dist-rep, iqn: null, vfsType: glusterfs, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 4799fa3d >2012-11-08 07:03:54,792 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-4-thread-50) [113ca761] FINISH, ConnectStorageServerVDSCommand, return: {0d9a1047-add1-41b9-911c-bf284b6fbfb2=0}, log id: 4799fa3d >2012-11-08 07:03:54,793 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-4-thread-49) [3f59adf9] FINISH, ConnectStorageServerVDSCommand, return: {0d9a1047-add1-41b9-911c-bf284b6fbfb2=0}, log id: 2272ecb2 >2012-11-08 07:03:54,794 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (pool-4-thread-48) [72e2c6f5] START, ActivateStorageDomainVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null, storageDomainId = e0cce407-ebea-4322-b47e-7aceb73a7a6f), log id: 5e3a1eae >2012-11-08 07:03:57,222 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (pool-4-thread-48) [72e2c6f5] FINISH, ActivateStorageDomainVDSCommand, log id: 5e3a1eae >2012-11-08 07:03:57,222 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (pool-4-thread-48) [72e2c6f5] ActivateStorage Domain. After Connect all hosts to pool. Time:11/8/12 7:03 AM >2012-11-08 07:03:57,276 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.RefreshStoragePoolVDSCommand] (pool-4-thread-47) [72e2c6f5] START, RefreshStoragePoolVDSCommand(HostName = rhs-gp-srv4.lab.eng.blr.redhat.com, HostId = d2bd32d4-2992-11e2-ab59-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, masterStorageDomainId=a02e96be-cf4c-4d0f-8368-fd9d4aa659bf, msterVersion=1), log id: 21495cac >2012-11-08 07:03:57,287 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.RefreshStoragePoolVDSCommand] (pool-4-thread-49) [72e2c6f5] START, RefreshStoragePoolVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, masterStorageDomainId=a02e96be-cf4c-4d0f-8368-fd9d4aa659bf, msterVersion=1), log id: 57633146 >2012-11-08 07:03:57,765 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.RefreshStoragePoolVDSCommand] (pool-4-thread-49) [72e2c6f5] FINISH, RefreshStoragePoolVDSCommand, log id: 57633146 >2012-11-08 07:03:57,765 INFO [org.ovirt.engine.core.bll.storage.RefreshPoolSingleAsyncOperation] (pool-4-thread-49) [72e2c6f5] Refreshed vds rhs-gp-srv11.lab.eng.blr.redhat.com in pool RHS-REBALANCE >2012-11-08 07:03:59,325 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-4-thread-47) [72e2c6f5] Failed in RefreshStoragePoolVDS method >2012-11-08 07:03:59,326 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-4-thread-47) [72e2c6f5] Error code StorageDomainAccessError and error message VDSGenericException: VDSErrorException: Failed to RefreshStoragePoolVDS, error = Domain is either partially accessible or entirely inaccessible: ('a02e96be-cf4c-4d0f-8368-fd9d4aa659bf',) >2012-11-08 07:03:59,327 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-4-thread-47) [72e2c6f5] Command org.ovirt.engine.core.vdsbroker.vdsbroker.RefreshStoragePoolVDSCommand return value > Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc >mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc >mCode 379 >mMessage Domain is either partially accessible or entirely inaccessible: ('a02e96be-cf4c-4d0f-8368-fd9d4aa659bf',) > > >2012-11-08 07:03:59,327 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-4-thread-47) [72e2c6f5] HostName = rhs-gp-srv4.lab.eng.blr.redhat.com >2012-11-08 07:03:59,327 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-4-thread-47) [72e2c6f5] Command RefreshStoragePoolVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to RefreshStoragePoolVDS, error = Domain is either partially accessible or entirely inaccessible: ('a02e96be-cf4c-4d0f-8368-fd9d4aa659bf',) >2012-11-08 07:03:59,327 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.RefreshStoragePoolVDSCommand] (pool-4-thread-47) [72e2c6f5] FINISH, RefreshStoragePoolVDSCommand, log id: 21495cac >2012-11-08 07:03:59,327 ERROR [org.ovirt.engine.core.bll.storage.RefreshPoolSingleAsyncOperation] (pool-4-thread-47) [72e2c6f5] Failed to connect/refresh storagePool. Host rhs-gp-srv4.lab.eng.blr.redhat.com to storage pool RHS-REBALANCE. Exception: {3}: org.ovirt.engine.core.common.errors.VdcBLLException: VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to RefreshStoragePoolVDS, error = Domain is either partially accessible or entirely inaccessible: ('a02e96be-cf4c-4d0f-8368-fd9d4aa659bf',) > at org.ovirt.engine.core.bll.VdsHandler.handleVdsResult(VdsHandler.java:212) [engine-bll.jar:] > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) [engine-bll.jar:] > at org.ovirt.engine.core.bll.storage.RefreshPoolSingleAsyncOperation.Execute(RefreshPoolSingleAsyncOperation.java:52) [engine-bll.jar:] > at org.ovirt.engine.core.utils.SyncronizeNumberOfAsyncOperations$AsyncOpThread.run(SyncronizeNumberOfAsyncOperations.java:30) [engine-utils.jar:] > at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalWrapperRunnable.run(ThreadPoolUtil.java:64) [engine-utils.jar:] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_09-icedtea] > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) [rt.jar:1.7.0_09-icedtea] > at java.util.concurrent.FutureTask.run(FutureTask.java:166) [rt.jar:1.7.0_09-icedtea] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [rt.jar:1.7.0_09-icedtea] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [rt.jar:1.7.0_09-icedtea] > at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_09-icedtea] > >2012-11-08 07:03:59,330 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (pool-4-thread-48) [72e2c6f5] ActivateStorage Domain. After change storage pool status in vds. Time:11/8/12 7:03 AM >2012-11-08 07:04:07,703 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-79) domain a02e96be-cf4c-4d0f-8368-fd9d4aa659bf:REBDOMAIN in problem. vds: rhs-gp-srv4.lab.eng.blr.redhat.com >2012-11-08 07:04:07,704 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-79) domain e0cce407-ebea-4322-b47e-7aceb73a7a6f:DISREP in problem. vds: rhs-gp-srv4.lab.eng.blr.redhat.com >2012-11-08 07:04:13,768 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-62) Failed to decrypt Data must not be longer than 256 bytes >2012-11-08 07:04:13,768 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-92) Failed to decrypt Data must not be longer than 256 bytes >2012-11-08 07:04:38,464 INFO [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (ajp-/127.0.0.1:8702-9) [482d03d5] Lock Acquired to object EngineLock [exclusiveLocks= key: disrep1 value: VM_NAME >, sharedLocks= ] >2012-11-08 07:04:38,481 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-9) [482d03d5] START, IsValidVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 23787222 >2012-11-08 07:04:38,482 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-9) [482d03d5] FINISH, IsValidVDSCommand, return: true, log id: 23787222 >2012-11-08 07:04:38,486 INFO [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (ajp-/127.0.0.1:8702-9) [482d03d5] Running command: AddVmFromScratchCommand internal: false. Entities affected : ID: 6cea47a6-2976-11e2-ad99-52540035ed48 Type: VdsGroups >2012-11-08 07:04:38,487 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-9) [482d03d5] START, IsValidVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 5d4c1a3a >2012-11-08 07:04:38,488 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-9) [482d03d5] FINISH, IsValidVDSCommand, return: true, log id: 5d4c1a3a >2012-11-08 07:04:38,503 INFO [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (ajp-/127.0.0.1:8702-9) [482d03d5] Lock freed to object EngineLock [exclusiveLocks= key: disrep1 value: VM_NAME >, sharedLocks= ] >2012-11-08 07:04:43,538 INFO [org.ovirt.engine.core.bll.AddVmInterfaceCommand] (ajp-/127.0.0.1:8702-10) [3886867f] Running command: AddVmInterfaceCommand internal: false. Entities affected : ID: 661d6af9-9024-4744-91b1-258e38466815 Type: VM >2012-11-08 07:04:43,539 INFO [org.ovirt.engine.core.bll.MacPoolManager] (ajp-/127.0.0.1:8702-10) [3886867f] MacPoolManager::allocateNewMac entered >2012-11-08 07:04:43,539 INFO [org.ovirt.engine.core.bll.MacPoolManager] (ajp-/127.0.0.1:8702-10) [3886867f] MacPoolManager::allocateNewMac allocated mac = 00:1a:4a:46:22:b1 >2012-11-08 07:04:43,573 INFO [org.ovirt.engine.core.bll.ActivateDeactivateVmNicCommand] (ajp-/127.0.0.1:8702-10) [53cb8f76] Running command: ActivateDeactivateVmNicCommand internal: true. Entities affected : ID: 661d6af9-9024-4744-91b1-258e38466815 Type: VM >2012-11-08 07:04:51,626 INFO [org.ovirt.engine.core.bll.AddDiskCommand] (ajp-/127.0.0.1:8702-1) [64a103bc] Lock Acquired to object EngineLock [exclusiveLocks= key: 661d6af9-9024-4744-91b1-258e38466815 value: VM_DISK_BOOT >, sharedLocks= key: 661d6af9-9024-4744-91b1-258e38466815 value: VM >] >2012-11-08 07:04:51,629 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-1) [64a103bc] START, IsValidVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 4801eb76 >2012-11-08 07:04:51,629 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-1) [64a103bc] FINISH, IsValidVDSCommand, return: true, log id: 4801eb76 >2012-11-08 07:04:51,635 INFO [org.ovirt.engine.core.bll.AddDiskCommand] (ajp-/127.0.0.1:8702-1) [64a103bc] Running command: AddDiskCommand internal: false. Entities affected : ID: 661d6af9-9024-4744-91b1-258e38466815 Type: VM, ID: e0cce407-ebea-4322-b47e-7aceb73a7a6f Type: Storage >2012-11-08 07:04:51,661 INFO [org.ovirt.engine.core.bll.AddImageFromScratchCommand] (ajp-/127.0.0.1:8702-1) [12b7ce92] Running command: AddImageFromScratchCommand internal: true. Entities affected : ID: e0cce407-ebea-4322-b47e-7aceb73a7a6f Type: Storage >2012-11-08 07:04:51,668 INFO [org.ovirt.engine.core.bll.AddImageFromScratchCommand] (ajp-/127.0.0.1:8702-1) [12b7ce92] Lock freed to object EngineLock [exclusiveLocks= key: 661d6af9-9024-4744-91b1-258e38466815 value: VM_DISK_BOOT >, sharedLocks= key: 661d6af9-9024-4744-91b1-258e38466815 value: VM >] >2012-11-08 07:04:51,669 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (ajp-/127.0.0.1:8702-1) [12b7ce92] START, CreateImageVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = 3.1, storageDomainId = e0cce407-ebea-4322-b47e-7aceb73a7a6f, imageGroupId = 23ec97a0-b4ec-4f07-8b26-aa27597d9f1a, imageSizeInBytes = 5368709120, volumeFormat = RAW, newImageId = 9ea10c33-7706-436f-a55e-ef6ef95a4f48, newImageDescription = ), log id: 178df96e >2012-11-08 07:04:51,669 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (ajp-/127.0.0.1:8702-1) [12b7ce92] -- CreateImageVDSCommand::ExecuteIrsBrokerCommand: calling 'createVolume' with two new parameters: description and UUID >2012-11-08 07:04:51,669 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (ajp-/127.0.0.1:8702-1) [12b7ce92] -- createVolume parameters: > sdUUID=e0cce407-ebea-4322-b47e-7aceb73a7a6f > spUUID=3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 > imgGUID=23ec97a0-b4ec-4f07-8b26-aa27597d9f1a > size=5,368,709,120 bytes > volFormat=RAW > volType=Preallocated > volUUID=9ea10c33-7706-436f-a55e-ef6ef95a4f48 > descr= > srcImgGUID=00000000-0000-0000-0000-000000000000 > srcVolUUID=00000000-0000-0000-0000-000000000000 > >2012-11-08 07:04:51,738 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (ajp-/127.0.0.1:8702-1) [12b7ce92] FINISH, CreateImageVDSCommand, return: 9ea10c33-7706-436f-a55e-ef6ef95a4f48, log id: 178df96e >2012-11-08 07:04:51,740 INFO [org.ovirt.engine.core.bll.EntityAsyncTask] (ajp-/127.0.0.1:8702-1) [12b7ce92] EntityAsyncTask::Adding EntityMultiAsyncTasks object for entity 23ec97a0-b4ec-4f07-8b26-aa27597d9f1a >2012-11-08 07:04:51,740 INFO [org.ovirt.engine.core.bll.EntityMultiAsyncTasks] (ajp-/127.0.0.1:8702-1) [12b7ce92] EntityMultiAsyncTasks::AttachTask: Attaching task 69e0e045-6a11-4bd2-ad73-554d671a3366 to entity 23ec97a0-b4ec-4f07-8b26-aa27597d9f1a. >2012-11-08 07:04:51,742 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskUtils] (ajp-/127.0.0.1:8702-1) [12b7ce92] Adding task 69e0e045-6a11-4bd2-ad73-554d671a3366 to DataBase >2012-11-08 07:04:51,773 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (ajp-/127.0.0.1:8702-1) [12b7ce92] Adding task 69e0e045-6a11-4bd2-ad73-554d671a3366 (Parent Command AddDisk, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters), polling hasn't started yet.. >2012-11-08 07:04:51,781 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (ajp-/127.0.0.1:8702-1) [12b7ce92] BaseAsyncTask::StartPollingTask: Starting to poll task 69e0e045-6a11-4bd2-ad73-554d671a3366. >2012-11-08 07:04:51,781 INFO [org.ovirt.engine.core.bll.EntityMultiAsyncTasks] (ajp-/127.0.0.1:8702-1) [12b7ce92] EntityMultiAsyncTasks::StartPollingTask: Current Action Type for entity 23ec97a0-b4ec-4f07-8b26-aa27597d9f1a is AddDisk (determined by task 69e0e045-6a11-4bd2-ad73-554d671a3366) >2012-11-08 07:04:57,075 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-74) Polling and updating Async Tasks: 1 tasks, 1 tasks to poll now >2012-11-08 07:04:57,118 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (QuartzScheduler_Worker-74) SPMAsyncTask::PollTask: Polling task 69e0e045-6a11-4bd2-ad73-554d671a3366 (Parent Command AddDisk, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status running. >2012-11-08 07:04:57,118 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-74) Finished polling Tasks, will poll again in 10 seconds. >2012-11-08 07:05:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-83) Autorecovering hosts is disabled, skipping >2012-11-08 07:05:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-83) Autorecovering storage domains is disabled, skipping >2012-11-08 07:05:37,268 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (QuartzScheduler_Worker-49) SPMAsyncTask::PollTask: Polling task 69e0e045-6a11-4bd2-ad73-554d671a3366 (Parent Command AddDisk, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status finished, result 'success'. >2012-11-08 07:05:37,272 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (QuartzScheduler_Worker-49) BaseAsyncTask::OnTaskEndSuccess: Task 69e0e045-6a11-4bd2-ad73-554d671a3366 (Parent Command AddDisk, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended successfully. >2012-11-08 07:05:37,272 INFO [org.ovirt.engine.core.bll.EntityAsyncTask] (QuartzScheduler_Worker-49) EntityAsyncTask::EndActionIfNecessary: All tasks of entity 23ec97a0-b4ec-4f07-8b26-aa27597d9f1a has ended -> executing EndAction >2012-11-08 07:05:37,272 INFO [org.ovirt.engine.core.bll.EntityAsyncTask] (QuartzScheduler_Worker-49) EntityAsyncTask::EndAction: Ending action for 1 tasks (entity ID: 23ec97a0-b4ec-4f07-8b26-aa27597d9f1a): calling EndAction for action type AddDisk. >2012-11-08 07:05:37,272 INFO [org.ovirt.engine.core.bll.EntityAsyncTask] (pool-4-thread-47) EntityAsyncTask::EndCommandAction [within thread]context: Attempting to EndAction AddDisk >2012-11-08 07:05:37,279 INFO [org.ovirt.engine.core.bll.AddDiskCommand] (pool-4-thread-47) [64a103bc] Ending command successfully: org.ovirt.engine.core.bll.AddDiskCommand >2012-11-08 07:05:37,279 INFO [org.ovirt.engine.core.bll.AddImageFromScratchCommand] (pool-4-thread-47) [12b7ce92] Ending command successfully: org.ovirt.engine.core.bll.AddImageFromScratchCommand >2012-11-08 07:05:37,286 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (pool-4-thread-47) [12b7ce92] START, GetImageInfoVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null, storageDomainId = e0cce407-ebea-4322-b47e-7aceb73a7a6f, imageGroupId = 23ec97a0-b4ec-4f07-8b26-aa27597d9f1a, imageId = 9ea10c33-7706-436f-a55e-ef6ef95a4f48), log id: 4479b71b >2012-11-08 07:05:37,345 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (pool-4-thread-47) [12b7ce92] FINISH, GetImageInfoVDSCommand, return: org.ovirt.engine.core.common.businessentities.DiskImage@e79ddcc2, log id: 4479b71b >2012-11-08 07:05:37,362 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UpdateVMVDSCommand] (pool-4-thread-47) START, UpdateVMVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null, storageDomainId = 00000000-0000-0000-0000-000000000000, infoDictionary.size = 1), log id: 13a1a55d >2012-11-08 07:05:37,406 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UpdateVMVDSCommand] (pool-4-thread-47) FINISH, UpdateVMVDSCommand, log id: 13a1a55d >2012-11-08 07:05:37,428 INFO [org.ovirt.engine.core.bll.EntityAsyncTask] (pool-4-thread-47) EntityAsyncTask::HandleEndActionResult [within thread]: EndAction for action type AddDisk completed, handling the result. >2012-11-08 07:05:37,428 INFO [org.ovirt.engine.core.bll.EntityAsyncTask] (pool-4-thread-47) EntityAsyncTask::HandleEndActionResult [within thread]: EndAction for action type AddDisk succeeded, clearing tasks. >2012-11-08 07:05:37,430 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (pool-4-thread-47) SPMAsyncTask::ClearAsyncTask: Attempting to clear task 69e0e045-6a11-4bd2-ad73-554d671a3366 >2012-11-08 07:05:37,431 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (pool-4-thread-47) START, SPMClearTaskVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null, taskId = 69e0e045-6a11-4bd2-ad73-554d671a3366), log id: 5ef44e05 >2012-11-08 07:05:37,447 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (pool-4-thread-47) START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=69e0e045-6a11-4bd2-ad73-554d671a3366), log id: 7ed30aa8 >2012-11-08 07:05:37,474 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (pool-4-thread-47) FINISH, HSMClearTaskVDSCommand, log id: 7ed30aa8 >2012-11-08 07:05:37,474 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (pool-4-thread-47) FINISH, SPMClearTaskVDSCommand, log id: 5ef44e05 >2012-11-08 07:05:37,476 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (pool-4-thread-47) BaseAsyncTask::RemoveTaskFromDB: Removed task 69e0e045-6a11-4bd2-ad73-554d671a3366 from DataBase >2012-11-08 07:05:37,476 INFO [org.ovirt.engine.core.bll.EntityAsyncTask] (pool-4-thread-47) EntityAsyncTask::HandleEndActionResult [within thread]: Removing EntityMultiAsyncTasks object for entity 23ec97a0-b4ec-4f07-8b26-aa27597d9f1a >2012-11-08 07:05:45,678 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-65) Failed to decrypt Data must not be longer than 256 bytes >2012-11-08 07:05:45,678 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-61) Failed to decrypt Data must not be longer than 256 bytes >2012-11-08 07:05:47,532 INFO [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (ajp-/127.0.0.1:8702-6) [ba3c11a] Lock Acquired to object EngineLock [exclusiveLocks= key: b3 value: VM_NAME >, sharedLocks= ] >2012-11-08 07:05:47,543 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-6) [ba3c11a] START, IsValidVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 47f14807 >2012-11-08 07:05:47,543 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-6) [ba3c11a] FINISH, IsValidVDSCommand, return: true, log id: 47f14807 >2012-11-08 07:05:47,547 INFO [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (ajp-/127.0.0.1:8702-6) [ba3c11a] Running command: AddVmFromScratchCommand internal: false. Entities affected : ID: 6cea47a6-2976-11e2-ad99-52540035ed48 Type: VdsGroups >2012-11-08 07:05:47,547 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-6) [ba3c11a] START, IsValidVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 1966aeaf >2012-11-08 07:05:47,548 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-6) [ba3c11a] FINISH, IsValidVDSCommand, return: true, log id: 1966aeaf >2012-11-08 07:05:47,558 INFO [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (ajp-/127.0.0.1:8702-6) [ba3c11a] Lock freed to object EngineLock [exclusiveLocks= key: b3 value: VM_NAME >, sharedLocks= ] >2012-11-08 07:06:14,293 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-11) Failed to decrypt Data must not be longer than 256 bytes >2012-11-08 07:06:14,293 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-15) Failed to decrypt Data must not be longer than 256 bytes >2012-11-08 07:06:18,351 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-27) Failed to decrypt Data must not be longer than 256 bytes >2012-11-08 07:06:18,361 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-23) Failed to decrypt Data must start with zero >2012-11-08 07:06:39,011 INFO [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (ajp-/127.0.0.1:8702-9) [48db5e66] Lock Acquired to object EngineLock [exclusiveLocks= key: disrep2 value: VM_NAME >, sharedLocks= ] >2012-11-08 07:06:39,022 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-9) [48db5e66] START, IsValidVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 7a2c0598 >2012-11-08 07:06:39,022 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-9) [48db5e66] FINISH, IsValidVDSCommand, return: true, log id: 7a2c0598 >2012-11-08 07:06:39,026 INFO [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (ajp-/127.0.0.1:8702-9) [48db5e66] Running command: AddVmFromScratchCommand internal: false. Entities affected : ID: 6cea47a6-2976-11e2-ad99-52540035ed48 Type: VdsGroups >2012-11-08 07:06:39,027 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-9) [48db5e66] START, IsValidVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 684bd0d >2012-11-08 07:06:39,027 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-9) [48db5e66] FINISH, IsValidVDSCommand, return: true, log id: 684bd0d >2012-11-08 07:06:39,037 INFO [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (ajp-/127.0.0.1:8702-9) [48db5e66] Lock freed to object EngineLock [exclusiveLocks= key: disrep2 value: VM_NAME >, sharedLocks= ] >2012-11-08 07:06:56,274 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-91) Setting new tasks map. The map contains now 0 tasks >2012-11-08 07:06:56,274 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-91) Cleared all tasks of pool 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6. >2012-11-08 07:06:56,284 INFO [org.ovirt.engine.core.bll.AddDiskCommand] (ajp-/127.0.0.1:8702-4) [1174ee8] Lock Acquired to object EngineLock [exclusiveLocks= key: 8a0d2e26-440c-4d6c-91d3-1fc4d9b75990 value: VM_DISK_BOOT >, sharedLocks= key: 8a0d2e26-440c-4d6c-91d3-1fc4d9b75990 value: VM >] >2012-11-08 07:06:56,287 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-4) [1174ee8] START, IsValidVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 21cb7f63 >2012-11-08 07:06:56,287 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-4) [1174ee8] FINISH, IsValidVDSCommand, return: true, log id: 21cb7f63 >2012-11-08 07:06:56,293 INFO [org.ovirt.engine.core.bll.AddDiskCommand] (ajp-/127.0.0.1:8702-4) [1174ee8] Running command: AddDiskCommand internal: false. Entities affected : ID: 8a0d2e26-440c-4d6c-91d3-1fc4d9b75990 Type: VM, ID: e0cce407-ebea-4322-b47e-7aceb73a7a6f Type: Storage >2012-11-08 07:06:56,296 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-91) Could not find vds that is spm and non-operational. >2012-11-08 07:06:56,316 INFO [org.ovirt.engine.core.bll.AddImageFromScratchCommand] (ajp-/127.0.0.1:8702-4) [51d2d5cd] Running command: AddImageFromScratchCommand internal: true. Entities affected : ID: e0cce407-ebea-4322-b47e-7aceb73a7a6f Type: Storage >2012-11-08 07:06:56,322 INFO [org.ovirt.engine.core.bll.AddImageFromScratchCommand] (ajp-/127.0.0.1:8702-4) [51d2d5cd] Lock freed to object EngineLock [exclusiveLocks= key: 8a0d2e26-440c-4d6c-91d3-1fc4d9b75990 value: VM_DISK_BOOT >, sharedLocks= key: 8a0d2e26-440c-4d6c-91d3-1fc4d9b75990 value: VM >] >2012-11-08 07:06:56,323 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (ajp-/127.0.0.1:8702-4) [51d2d5cd] START, CreateImageVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = 3.1, storageDomainId = e0cce407-ebea-4322-b47e-7aceb73a7a6f, imageGroupId = 8e861032-8233-4a00-9d20-49c3fb133088, imageSizeInBytes = 5368709120, volumeFormat = RAW, newImageId = d58cfacd-60a5-42df-a4fc-4f7c07a89eb5, newImageDescription = ), log id: 46c00477 >2012-11-08 07:06:56,324 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (ajp-/127.0.0.1:8702-4) [51d2d5cd] -- CreateImageVDSCommand::ExecuteIrsBrokerCommand: calling 'createVolume' with two new parameters: description and UUID >2012-11-08 07:06:56,324 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (ajp-/127.0.0.1:8702-4) [51d2d5cd] -- createVolume parameters: > sdUUID=e0cce407-ebea-4322-b47e-7aceb73a7a6f > spUUID=3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 > imgGUID=8e861032-8233-4a00-9d20-49c3fb133088 > size=5,368,709,120 bytes > volFormat=RAW > volType=Preallocated > volUUID=d58cfacd-60a5-42df-a4fc-4f7c07a89eb5 > descr= > srcImgGUID=00000000-0000-0000-0000-000000000000 > srcVolUUID=00000000-0000-0000-0000-000000000000 > >2012-11-08 07:06:56,399 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (ajp-/127.0.0.1:8702-4) [51d2d5cd] FINISH, CreateImageVDSCommand, return: d58cfacd-60a5-42df-a4fc-4f7c07a89eb5, log id: 46c00477 >2012-11-08 07:06:56,401 INFO [org.ovirt.engine.core.bll.EntityAsyncTask] (ajp-/127.0.0.1:8702-4) [51d2d5cd] EntityAsyncTask::Adding EntityMultiAsyncTasks object for entity 8e861032-8233-4a00-9d20-49c3fb133088 >2012-11-08 07:06:56,401 INFO [org.ovirt.engine.core.bll.EntityMultiAsyncTasks] (ajp-/127.0.0.1:8702-4) [51d2d5cd] EntityMultiAsyncTasks::AttachTask: Attaching task a15b3f37-3d1a-45f2-8772-ff73c45bba35 to entity 8e861032-8233-4a00-9d20-49c3fb133088. >2012-11-08 07:06:56,402 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskUtils] (ajp-/127.0.0.1:8702-4) [51d2d5cd] Adding task a15b3f37-3d1a-45f2-8772-ff73c45bba35 to DataBase >2012-11-08 07:06:56,419 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (ajp-/127.0.0.1:8702-4) [51d2d5cd] Adding task a15b3f37-3d1a-45f2-8772-ff73c45bba35 (Parent Command AddDisk, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters), polling hasn't started yet.. >2012-11-08 07:06:56,425 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (ajp-/127.0.0.1:8702-4) [51d2d5cd] BaseAsyncTask::StartPollingTask: Starting to poll task a15b3f37-3d1a-45f2-8772-ff73c45bba35. >2012-11-08 07:06:56,425 INFO [org.ovirt.engine.core.bll.EntityMultiAsyncTasks] (ajp-/127.0.0.1:8702-4) [51d2d5cd] EntityMultiAsyncTasks::StartPollingTask: Current Action Type for entity 8e861032-8233-4a00-9d20-49c3fb133088 is AddDisk (determined by task a15b3f37-3d1a-45f2-8772-ff73c45bba35) >2012-11-08 07:06:57,278 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-96) Polling and updating Async Tasks: 1 tasks, 1 tasks to poll now >2012-11-08 07:06:57,320 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (QuartzScheduler_Worker-96) SPMAsyncTask::PollTask: Polling task a15b3f37-3d1a-45f2-8772-ff73c45bba35 (Parent Command AddDisk, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status running. >2012-11-08 07:06:57,320 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-96) Finished polling Tasks, will poll again in 10 seconds. >2012-11-08 07:07:04,890 INFO [org.ovirt.engine.core.bll.AddVmInterfaceCommand] (ajp-/127.0.0.1:8702-12) [40f8066] Running command: AddVmInterfaceCommand internal: false. Entities affected : ID: 8a0d2e26-440c-4d6c-91d3-1fc4d9b75990 Type: VM >2012-11-08 07:07:04,891 INFO [org.ovirt.engine.core.bll.MacPoolManager] (ajp-/127.0.0.1:8702-12) [40f8066] MacPoolManager::allocateNewMac entered >2012-11-08 07:07:04,891 INFO [org.ovirt.engine.core.bll.MacPoolManager] (ajp-/127.0.0.1:8702-12) [40f8066] MacPoolManager::allocateNewMac allocated mac = 00:1a:4a:46:22:4f >2012-11-08 07:07:04,916 INFO [org.ovirt.engine.core.bll.ActivateDeactivateVmNicCommand] (ajp-/127.0.0.1:8702-12) [58b5d2d8] Running command: ActivateDeactivateVmNicCommand internal: true. Entities affected : ID: 8a0d2e26-440c-4d6c-91d3-1fc4d9b75990 Type: VM >2012-11-08 07:07:28,120 INFO [org.ovirt.engine.core.bll.AddDiskCommand] (ajp-/127.0.0.1:8702-1) Lock Acquired to object EngineLock [exclusiveLocks= key: 7fa6f92f-755f-44c3-8f35-b4c56410c87b value: VM_DISK_BOOT >, sharedLocks= key: 7fa6f92f-755f-44c3-8f35-b4c56410c87b value: VM >] >2012-11-08 07:07:28,136 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-1) START, IsValidVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 797fcf69 >2012-11-08 07:07:28,137 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-1) FINISH, IsValidVDSCommand, return: true, log id: 797fcf69 >2012-11-08 07:07:28,152 INFO [org.ovirt.engine.core.bll.AddDiskCommand] (pool-4-thread-50) [a6507af] Running command: AddDiskCommand internal: false. Entities affected : ID: 7fa6f92f-755f-44c3-8f35-b4c56410c87b Type: VM, ID: a02e96be-cf4c-4d0f-8368-fd9d4aa659bf Type: Storage >2012-11-08 07:07:28,170 INFO [org.ovirt.engine.core.bll.AddImageFromScratchCommand] (pool-4-thread-50) [4a7c974b] Running command: AddImageFromScratchCommand internal: true. Entities affected : ID: a02e96be-cf4c-4d0f-8368-fd9d4aa659bf Type: Storage >2012-11-08 07:07:28,177 INFO [org.ovirt.engine.core.bll.AddImageFromScratchCommand] (pool-4-thread-50) [4a7c974b] Lock freed to object EngineLock [exclusiveLocks= key: 7fa6f92f-755f-44c3-8f35-b4c56410c87b value: VM_DISK_BOOT >, sharedLocks= key: 7fa6f92f-755f-44c3-8f35-b4c56410c87b value: VM >] >2012-11-08 07:07:28,178 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (pool-4-thread-50) [4a7c974b] START, CreateImageVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = 3.1, storageDomainId = a02e96be-cf4c-4d0f-8368-fd9d4aa659bf, imageGroupId = 7ca356eb-9853-4076-9347-172bccbf8d32, imageSizeInBytes = 5368709120, volumeFormat = RAW, newImageId = a6d0bc63-e895-451d-9662-b9a2e419891f, newImageDescription = ), log id: 1a4ab013 >2012-11-08 07:07:28,178 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (pool-4-thread-50) [4a7c974b] -- CreateImageVDSCommand::ExecuteIrsBrokerCommand: calling 'createVolume' with two new parameters: description and UUID >2012-11-08 07:07:28,178 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (pool-4-thread-50) [4a7c974b] -- createVolume parameters: > sdUUID=a02e96be-cf4c-4d0f-8368-fd9d4aa659bf > spUUID=3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 > imgGUID=7ca356eb-9853-4076-9347-172bccbf8d32 > size=5,368,709,120 bytes > volFormat=RAW > volType=Preallocated > volUUID=a6d0bc63-e895-451d-9662-b9a2e419891f > descr= > srcImgGUID=00000000-0000-0000-0000-000000000000 > srcVolUUID=00000000-0000-0000-0000-000000000000 > >2012-11-08 07:07:28,247 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (pool-4-thread-50) [4a7c974b] FINISH, CreateImageVDSCommand, return: a6d0bc63-e895-451d-9662-b9a2e419891f, log id: 1a4ab013 >2012-11-08 07:07:28,250 INFO [org.ovirt.engine.core.bll.EntityAsyncTask] (pool-4-thread-50) [4a7c974b] EntityAsyncTask::Adding EntityMultiAsyncTasks object for entity 7ca356eb-9853-4076-9347-172bccbf8d32 >2012-11-08 07:07:28,250 INFO [org.ovirt.engine.core.bll.EntityMultiAsyncTasks] (pool-4-thread-50) [4a7c974b] EntityMultiAsyncTasks::AttachTask: Attaching task a9c01093-106f-40d7-992c-e9d3bde0169c to entity 7ca356eb-9853-4076-9347-172bccbf8d32. >2012-11-08 07:07:28,251 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskUtils] (pool-4-thread-50) [4a7c974b] Adding task a9c01093-106f-40d7-992c-e9d3bde0169c to DataBase >2012-11-08 07:07:28,267 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (pool-4-thread-50) [4a7c974b] Adding task a9c01093-106f-40d7-992c-e9d3bde0169c (Parent Command AddDisk, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters), polling hasn't started yet.. >2012-11-08 07:07:28,277 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (pool-4-thread-50) [4a7c974b] BaseAsyncTask::StartPollingTask: Starting to poll task a9c01093-106f-40d7-992c-e9d3bde0169c. >2012-11-08 07:07:28,277 INFO [org.ovirt.engine.core.bll.EntityMultiAsyncTasks] (pool-4-thread-50) [4a7c974b] EntityMultiAsyncTasks::StartPollingTask: Current Action Type for entity 7ca356eb-9853-4076-9347-172bccbf8d32 is AddDisk (determined by task a9c01093-106f-40d7-992c-e9d3bde0169c) >2012-11-08 07:07:35,983 INFO [org.ovirt.engine.core.bll.AddVmInterfaceCommand] (ajp-/127.0.0.1:8702-14) [55944162] Running command: AddVmInterfaceCommand internal: false. Entities affected : ID: 7fa6f92f-755f-44c3-8f35-b4c56410c87b Type: VM >2012-11-08 07:07:35,984 INFO [org.ovirt.engine.core.bll.MacPoolManager] (ajp-/127.0.0.1:8702-14) [55944162] MacPoolManager::allocateNewMac entered >2012-11-08 07:07:35,984 INFO [org.ovirt.engine.core.bll.MacPoolManager] (ajp-/127.0.0.1:8702-14) [55944162] MacPoolManager::allocateNewMac allocated mac = 00:1a:4a:46:22:b2 >2012-11-08 07:07:36,029 INFO [org.ovirt.engine.core.bll.ActivateDeactivateVmNicCommand] (ajp-/127.0.0.1:8702-14) [35a2ff1] Running command: ActivateDeactivateVmNicCommand internal: true. Entities affected : ID: 7fa6f92f-755f-44c3-8f35-b4c56410c87b Type: VM >2012-11-08 07:07:37,437 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-72) Polling and updating Async Tasks: 2 tasks, 2 tasks to poll now >2012-11-08 07:07:37,473 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (QuartzScheduler_Worker-72) SPMAsyncTask::PollTask: Polling task a9c01093-106f-40d7-992c-e9d3bde0169c (Parent Command AddDisk, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status running. >2012-11-08 07:07:37,473 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-72) Finished polling Tasks, will poll again in 10 seconds. >2012-11-08 07:07:44,831 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-14) [7863592c] START, IsValidVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 43c692cf >2012-11-08 07:07:44,832 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-14) [7863592c] FINISH, IsValidVDSCommand, return: true, log id: 43c692cf >2012-11-08 07:07:44,840 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp-/127.0.0.1:8702-14) [7863592c] START, IsVmDuringInitiatingVDSCommand( vmId = 67bd2d56-8e0b-4328-bcfd-ea7f4758c885), log id: 28cf6ebe >2012-11-08 07:07:44,840 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp-/127.0.0.1:8702-14) [7863592c] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 28cf6ebe >2012-11-08 07:07:44,885 INFO [org.ovirt.engine.core.bll.RunVmOnceCommand] (ajp-/127.0.0.1:8702-14) [7863592c] Running command: RunVmOnceCommand internal: false. Entities affected : ID: 67bd2d56-8e0b-4328-bcfd-ea7f4758c885 Type: VM, ID: 67bd2d56-8e0b-4328-bcfd-ea7f4758c885 Type: VM >2012-11-08 07:07:44,958 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-14) [7863592c] START, CreateVmVDSCommand(HostName = rhs-gp-srv4.lab.eng.blr.redhat.com, HostId = d2bd32d4-2992-11e2-ab59-52540035ed48, vmId=67bd2d56-8e0b-4328-bcfd-ea7f4758c885, vm=org.ovirt.engine.core.common.businessentities.VM@57b7f5ce), log id: 69e084c9 >2012-11-08 07:07:44,968 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8702-14) [7863592c] START, CreateVDSCommand(HostName = rhs-gp-srv4.lab.eng.blr.redhat.com, HostId = d2bd32d4-2992-11e2-ab59-52540035ed48, vmId=67bd2d56-8e0b-4328-bcfd-ea7f4758c885, vm=org.ovirt.engine.core.common.businessentities.VM@57b7f5ce), log id: 5a9e87b >2012-11-08 07:07:45,020 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8702-14) [7863592c] org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand spiceSslCipherSuite=DEFAULT,memSize=512,kvmEnable=true,smp=1,emulatedMachine=rhel6.3.0,vmType=kvm,keyboardLayout=en-us,nice=0,display=qxl,smpCoresPerSocket=1,spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard,timeOffset=-43200,transparentHugePages=true,vmId=67bd2d56-8e0b-4328-bcfd-ea7f4758c885,devices=[Ljava.util.Map;@6250e528,acpiEnable=true,vmName=b1,cpuType=Nehalem,custom={viodiskcache=none} >2012-11-08 07:07:45,020 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8702-14) [7863592c] FINISH, CreateVDSCommand, log id: 5a9e87b >2012-11-08 07:07:45,021 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-14) [7863592c] IncreasePendingVms::CreateVmIncreasing vds rhs-gp-srv4.lab.eng.blr.redhat.com pending vcpu count, now 1. Vm: b1 >2012-11-08 07:07:45,026 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-14) [7863592c] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 69e084c9 >2012-11-08 07:07:46,207 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (QuartzScheduler_Worker-82) START, DestroyVDSCommand(HostName = rhs-gp-srv4.lab.eng.blr.redhat.com, HostId = d2bd32d4-2992-11e2-ab59-52540035ed48, vmId=67bd2d56-8e0b-4328-bcfd-ea7f4758c885, force=false, secondsToWait=0, gracefully=false), log id: 276adcd2 >2012-11-08 07:07:46,259 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (QuartzScheduler_Worker-82) FINISH, DestroyVDSCommand, log id: 276adcd2 >2012-11-08 07:07:46,278 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-82) Running on vds during rerun failed vm: null >2012-11-08 07:07:46,279 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-82) vm b1 running in db and not running in vds - add to rerun treatment. vds rhs-gp-srv4.lab.eng.blr.redhat.com >2012-11-08 07:07:46,284 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-82) Rerun vm 67bd2d56-8e0b-4328-bcfd-ea7f4758c885. Called from vds rhs-gp-srv4.lab.eng.blr.redhat.com >2012-11-08 07:07:46,310 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsDynamicDataVDSCommand] (pool-4-thread-50) START, UpdateVdsDynamicDataVDSCommand(HostName = rhs-gp-srv4.lab.eng.blr.redhat.com, HostId = d2bd32d4-2992-11e2-ab59-52540035ed48, vdsDynamic=org.ovirt.engine.core.common.businessentities.VdsDynamic@1763d912), log id: 521beda4 >2012-11-08 07:07:46,312 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsDynamicDataVDSCommand] (pool-4-thread-50) FINISH, UpdateVdsDynamicDataVDSCommand, log id: 521beda4 >2012-11-08 07:07:46,328 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (pool-4-thread-50) START, IsValidVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 68dcbc71 >2012-11-08 07:07:46,329 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (pool-4-thread-50) FINISH, IsValidVDSCommand, return: true, log id: 68dcbc71 >2012-11-08 07:07:46,351 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (pool-4-thread-50) START, IsVmDuringInitiatingVDSCommand( vmId = 67bd2d56-8e0b-4328-bcfd-ea7f4758c885), log id: 2ca8a5f3 >2012-11-08 07:07:46,351 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (pool-4-thread-50) FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 2ca8a5f3 >2012-11-08 07:07:46,389 INFO [org.ovirt.engine.core.bll.RunVmOnceCommand] (pool-4-thread-50) Running command: RunVmOnceCommand internal: false. Entities affected : ID: 67bd2d56-8e0b-4328-bcfd-ea7f4758c885 Type: VM, ID: 67bd2d56-8e0b-4328-bcfd-ea7f4758c885 Type: VM >2012-11-08 07:07:46,469 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (pool-4-thread-50) START, CreateVmVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=67bd2d56-8e0b-4328-bcfd-ea7f4758c885, vm=org.ovirt.engine.core.common.businessentities.VM@5eca52ac), log id: c42127f >2012-11-08 07:07:46,480 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (pool-4-thread-50) START, CreateVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=67bd2d56-8e0b-4328-bcfd-ea7f4758c885, vm=org.ovirt.engine.core.common.businessentities.VM@5eca52ac), log id: 22277680 >2012-11-08 07:07:46,525 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (pool-4-thread-50) org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand spiceSslCipherSuite=DEFAULT,memSize=512,kvmEnable=true,smp=1,emulatedMachine=rhel6.3.0,vmType=kvm,keyboardLayout=en-us,nice=0,display=qxl,smpCoresPerSocket=1,spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard,timeOffset=-43200,transparentHugePages=true,vmId=67bd2d56-8e0b-4328-bcfd-ea7f4758c885,devices=[Ljava.util.Map;@327f1324,acpiEnable=true,vmName=b1,cpuType=Nehalem,custom={viodiskcache=none} >2012-11-08 07:07:46,525 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (pool-4-thread-50) FINISH, CreateVDSCommand, log id: 22277680 >2012-11-08 07:07:46,526 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (pool-4-thread-50) IncreasePendingVms::CreateVmIncreasing vds rhs-gp-srv11.lab.eng.blr.redhat.com pending vcpu count, now 1. Vm: b1 >2012-11-08 07:07:46,528 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (pool-4-thread-50) FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: c42127f >2012-11-08 07:07:48,243 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (QuartzScheduler_Worker-89) START, DestroyVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=67bd2d56-8e0b-4328-bcfd-ea7f4758c885, force=false, secondsToWait=0, gracefully=false), log id: 7982b5a2 >2012-11-08 07:07:48,298 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (QuartzScheduler_Worker-89) FINISH, DestroyVDSCommand, log id: 7982b5a2 >2012-11-08 07:07:48,320 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-89) Running on vds during rerun failed vm: null >2012-11-08 07:07:48,320 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-89) vm b1 running in db and not running in vds - add to rerun treatment. vds rhs-gp-srv11.lab.eng.blr.redhat.com >2012-11-08 07:07:48,324 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-89) Rerun vm 67bd2d56-8e0b-4328-bcfd-ea7f4758c885. Called from vds rhs-gp-srv11.lab.eng.blr.redhat.com >2012-11-08 07:07:48,344 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsDynamicDataVDSCommand] (pool-4-thread-47) START, UpdateVdsDynamicDataVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vdsDynamic=org.ovirt.engine.core.common.businessentities.VdsDynamic@1944e16f), log id: 7a2bfe5d >2012-11-08 07:07:48,345 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsDynamicDataVDSCommand] (pool-4-thread-47) FINISH, UpdateVdsDynamicDataVDSCommand, log id: 7a2bfe5d >2012-11-08 07:07:48,367 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (pool-4-thread-47) START, IsValidVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 1365b53b >2012-11-08 07:07:48,368 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (pool-4-thread-47) FINISH, IsValidVDSCommand, return: true, log id: 1365b53b >2012-11-08 07:07:48,378 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (pool-4-thread-47) START, IsVmDuringInitiatingVDSCommand( vmId = 67bd2d56-8e0b-4328-bcfd-ea7f4758c885), log id: 559d833 >2012-11-08 07:07:48,378 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (pool-4-thread-47) FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 559d833 >2012-11-08 07:07:48,420 INFO [org.ovirt.engine.core.bll.VdsSelector] (pool-4-thread-47) VDS rhs-gp-srv11.lab.eng.blr.redhat.com 96739be4-2990-11e2-8577-52540035ed48 have failed running this VM in the current selection cycle VDS rhs-gp-srv4.lab.eng.blr.redhat.com d2bd32d4-2992-11e2-ab59-52540035ed48 is the same host the VM is currently running on VDS rhs-client11.lab.eng.blr.redhat.com 90b581c2-2977-11e2-82cd-52540035ed48 is not in up status or belongs to the VM's cluster >2012-11-08 07:07:48,420 WARN [org.ovirt.engine.core.bll.RunVmOnceCommand] (pool-4-thread-47) CanDoAction of action RunVmOnce failed. Reasons:VAR__ACTION__RUN,VAR__TYPE__VM,$VmName b1,VAR__ACTION__RUN,VAR__TYPE__VM,$VmName b1,VAR__ACTION__RUN,VAR__TYPE__VM,ACTION_TYPE_FAILED_VDS_VM_CLUSTER >2012-11-08 07:08:07,097 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-10) [68dc8aba] START, IsValidVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3de2ea99 >2012-11-08 07:08:07,097 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-10) [68dc8aba] FINISH, IsValidVDSCommand, return: true, log id: 3de2ea99 >2012-11-08 07:08:07,098 WARN [org.ovirt.engine.core.bll.RunVmOnceCommand] (ajp-/127.0.0.1:8702-10) [68dc8aba] CanDoAction of action RunVmOnce failed. Reasons:VAR__ACTION__RUN,VAR__TYPE__VM,ACTION_TYPE_FAILED_DISKS_ARE_LOCKED,$diskAliases disrep2_Disk1 >2012-11-08 07:08:41,419 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-88) Failed to decrypt Data must start with zero >2012-11-08 07:08:47,742 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (QuartzScheduler_Worker-98) SPMAsyncTask::PollTask: Polling task a15b3f37-3d1a-45f2-8772-ff73c45bba35 (Parent Command AddDisk, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status finished, result 'success'. >2012-11-08 07:08:47,747 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (QuartzScheduler_Worker-98) BaseAsyncTask::OnTaskEndSuccess: Task a15b3f37-3d1a-45f2-8772-ff73c45bba35 (Parent Command AddDisk, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended successfully. >2012-11-08 07:08:47,748 INFO [org.ovirt.engine.core.bll.EntityAsyncTask] (QuartzScheduler_Worker-98) EntityAsyncTask::EndActionIfNecessary: All tasks of entity 8e861032-8233-4a00-9d20-49c3fb133088 has ended -> executing EndAction >2012-11-08 07:08:47,748 INFO [org.ovirt.engine.core.bll.EntityAsyncTask] (QuartzScheduler_Worker-98) EntityAsyncTask::EndAction: Ending action for 1 tasks (entity ID: 8e861032-8233-4a00-9d20-49c3fb133088): calling EndAction for action type AddDisk. >2012-11-08 07:08:47,749 INFO [org.ovirt.engine.core.bll.EntityAsyncTask] (pool-4-thread-47) EntityAsyncTask::EndCommandAction [within thread]context: Attempting to EndAction AddDisk >2012-11-08 07:08:47,756 INFO [org.ovirt.engine.core.bll.AddDiskCommand] (pool-4-thread-47) [1174ee8] Ending command successfully: org.ovirt.engine.core.bll.AddDiskCommand >2012-11-08 07:08:47,756 INFO [org.ovirt.engine.core.bll.AddImageFromScratchCommand] (pool-4-thread-47) [51d2d5cd] Ending command successfully: org.ovirt.engine.core.bll.AddImageFromScratchCommand >2012-11-08 07:08:47,765 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (pool-4-thread-47) [51d2d5cd] START, GetImageInfoVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null, storageDomainId = e0cce407-ebea-4322-b47e-7aceb73a7a6f, imageGroupId = 8e861032-8233-4a00-9d20-49c3fb133088, imageId = d58cfacd-60a5-42df-a4fc-4f7c07a89eb5), log id: 681bc015 >2012-11-08 07:08:47,826 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (pool-4-thread-47) [51d2d5cd] FINISH, GetImageInfoVDSCommand, return: org.ovirt.engine.core.common.businessentities.DiskImage@fa19a582, log id: 681bc015 >2012-11-08 07:08:47,842 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UpdateVMVDSCommand] (pool-4-thread-47) START, UpdateVMVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null, storageDomainId = 00000000-0000-0000-0000-000000000000, infoDictionary.size = 1), log id: 704cc518 >2012-11-08 07:08:47,987 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UpdateVMVDSCommand] (pool-4-thread-47) FINISH, UpdateVMVDSCommand, log id: 704cc518 >2012-11-08 07:08:47,998 INFO [org.ovirt.engine.core.bll.EntityAsyncTask] (pool-4-thread-47) EntityAsyncTask::HandleEndActionResult [within thread]: EndAction for action type AddDisk completed, handling the result. >2012-11-08 07:08:47,998 INFO [org.ovirt.engine.core.bll.EntityAsyncTask] (pool-4-thread-47) EntityAsyncTask::HandleEndActionResult [within thread]: EndAction for action type AddDisk succeeded, clearing tasks. >2012-11-08 07:08:48,001 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (pool-4-thread-47) SPMAsyncTask::ClearAsyncTask: Attempting to clear task a15b3f37-3d1a-45f2-8772-ff73c45bba35 >2012-11-08 07:08:48,001 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (pool-4-thread-47) START, SPMClearTaskVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null, taskId = a15b3f37-3d1a-45f2-8772-ff73c45bba35), log id: 57dcbdd6 >2012-11-08 07:08:48,018 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (pool-4-thread-47) START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=a15b3f37-3d1a-45f2-8772-ff73c45bba35), log id: 1fb7799f >2012-11-08 07:08:48,248 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (pool-4-thread-47) FINISH, HSMClearTaskVDSCommand, log id: 1fb7799f >2012-11-08 07:08:48,248 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (pool-4-thread-47) FINISH, SPMClearTaskVDSCommand, log id: 57dcbdd6 >2012-11-08 07:08:48,251 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (pool-4-thread-47) BaseAsyncTask::RemoveTaskFromDB: Removed task a15b3f37-3d1a-45f2-8772-ff73c45bba35 from DataBase >2012-11-08 07:08:48,251 INFO [org.ovirt.engine.core.bll.EntityAsyncTask] (pool-4-thread-47) EntityAsyncTask::HandleEndActionResult [within thread]: Removing EntityMultiAsyncTasks object for entity 8e861032-8233-4a00-9d20-49c3fb133088 >2012-11-08 07:08:58,320 INFO [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (ajp-/127.0.0.1:8702-10) [753afe23] Lock Acquired to object EngineLock [exclusiveLocks= key: disrep3 value: VM_NAME >, sharedLocks= ] >2012-11-08 07:08:58,332 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-10) [753afe23] START, IsValidVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 165243b0 >2012-11-08 07:08:58,332 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-10) [753afe23] FINISH, IsValidVDSCommand, return: true, log id: 165243b0 >2012-11-08 07:08:58,336 INFO [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (ajp-/127.0.0.1:8702-10) [753afe23] Running command: AddVmFromScratchCommand internal: false. Entities affected : ID: 6cea47a6-2976-11e2-ad99-52540035ed48 Type: VdsGroups >2012-11-08 07:08:58,337 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-10) [753afe23] START, IsValidVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 55695a07 >2012-11-08 07:08:58,337 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-10) [753afe23] FINISH, IsValidVDSCommand, return: true, log id: 55695a07 >2012-11-08 07:08:58,350 INFO [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (ajp-/127.0.0.1:8702-10) [753afe23] Lock freed to object EngineLock [exclusiveLocks= key: disrep3 value: VM_NAME >, sharedLocks= ] >2012-11-08 07:09:07,705 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) starting ProcessDomainRecovery for domain a02e96be-cf4c-4d0f-8368-fd9d4aa659bf:REBDOMAIN >2012-11-08 07:09:07,736 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) vds rhs-gp-srv4.lab.eng.blr.redhat.com reported domain a02e96be-cf4c-4d0f-8368-fd9d4aa659bf:REBDOMAIN as in problem, moving the vds to status NonOperational >2012-11-08 07:09:07,768 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-31) [6143973] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: d2bd32d4-2992-11e2-ab59-52540035ed48 Type: VDS >2012-11-08 07:09:07,778 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-31) [6143973] START, SetVdsStatusVDSCommand(HostName = rhs-gp-srv4.lab.eng.blr.redhat.com, HostId = d2bd32d4-2992-11e2-ab59-52540035ed48, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 14c1225b >2012-11-08 07:09:07,788 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-31) [6143973] FINISH, SetVdsStatusVDSCommand, log id: 14c1225b >2012-11-08 07:09:07,799 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) [6143973] Clearing cache of pool: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 for problematic entities of VDS: rhs-gp-srv4.lab.eng.blr.redhat.com. >2012-11-08 07:09:07,805 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) [6143973] Domain a02e96be-cf4c-4d0f-8368-fd9d4aa659bf:REBDOMAIN recovered from problem. vds: rhs-gp-srv4.lab.eng.blr.redhat.com >2012-11-08 07:09:07,805 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) [6143973] Domain a02e96be-cf4c-4d0f-8368-fd9d4aa659bf:REBDOMAIN has recovered from problem. No active host in the DC is reporting it as problematic, so clearing the domain recovery timer. >2012-11-08 07:09:07,806 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) [6143973] Domain e0cce407-ebea-4322-b47e-7aceb73a7a6f:DISREP recovered from problem. vds: rhs-gp-srv4.lab.eng.blr.redhat.com >2012-11-08 07:09:07,806 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) [6143973] Domain e0cce407-ebea-4322-b47e-7aceb73a7a6f:DISREP has recovered from problem. No active host in the DC is reporting it as problematic, so clearing the domain recovery timer. >2012-11-08 07:09:08,783 INFO [org.ovirt.engine.core.bll.AddDiskCommand] (ajp-/127.0.0.1:8702-3) [5c108410] Lock Acquired to object EngineLock [exclusiveLocks= key: 0ed98fdf-e2d5-4bf6-b94a-3cba410c4c8f value: VM_DISK_BOOT >, sharedLocks= key: 0ed98fdf-e2d5-4bf6-b94a-3cba410c4c8f value: VM >] >2012-11-08 07:09:08,788 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-3) [5c108410] START, IsValidVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 24b595fd >2012-11-08 07:09:08,789 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-3) [5c108410] FINISH, IsValidVDSCommand, return: true, log id: 24b595fd >2012-11-08 07:09:08,797 INFO [org.ovirt.engine.core.bll.AddDiskCommand] (ajp-/127.0.0.1:8702-3) [5c108410] Running command: AddDiskCommand internal: false. Entities affected : ID: 0ed98fdf-e2d5-4bf6-b94a-3cba410c4c8f Type: VM, ID: e0cce407-ebea-4322-b47e-7aceb73a7a6f Type: Storage >2012-11-08 07:09:08,821 INFO [org.ovirt.engine.core.bll.AddImageFromScratchCommand] (ajp-/127.0.0.1:8702-3) [3f92cb5c] Running command: AddImageFromScratchCommand internal: true. Entities affected : ID: e0cce407-ebea-4322-b47e-7aceb73a7a6f Type: Storage >2012-11-08 07:09:08,827 INFO [org.ovirt.engine.core.bll.AddImageFromScratchCommand] (ajp-/127.0.0.1:8702-3) [3f92cb5c] Lock freed to object EngineLock [exclusiveLocks= key: 0ed98fdf-e2d5-4bf6-b94a-3cba410c4c8f value: VM_DISK_BOOT >, sharedLocks= key: 0ed98fdf-e2d5-4bf6-b94a-3cba410c4c8f value: VM >] >2012-11-08 07:09:08,828 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (ajp-/127.0.0.1:8702-3) [3f92cb5c] START, CreateImageVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = 3.1, storageDomainId = e0cce407-ebea-4322-b47e-7aceb73a7a6f, imageGroupId = 5cf389c5-51d0-46cc-9a5a-6042f734a901, imageSizeInBytes = 5368709120, volumeFormat = RAW, newImageId = 896e24bd-5bd3-4961-b8c2-f06267a723e2, newImageDescription = ), log id: 76d0f608 >2012-11-08 07:09:08,828 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (ajp-/127.0.0.1:8702-3) [3f92cb5c] -- CreateImageVDSCommand::ExecuteIrsBrokerCommand: calling 'createVolume' with two new parameters: description and UUID >2012-11-08 07:09:08,828 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (ajp-/127.0.0.1:8702-3) [3f92cb5c] -- createVolume parameters: > sdUUID=e0cce407-ebea-4322-b47e-7aceb73a7a6f > spUUID=3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 > imgGUID=5cf389c5-51d0-46cc-9a5a-6042f734a901 > size=5,368,709,120 bytes > volFormat=RAW > volType=Preallocated > volUUID=896e24bd-5bd3-4961-b8c2-f06267a723e2 > descr= > srcImgGUID=00000000-0000-0000-0000-000000000000 > srcVolUUID=00000000-0000-0000-0000-000000000000 > >2012-11-08 07:09:08,898 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (ajp-/127.0.0.1:8702-3) [3f92cb5c] FINISH, CreateImageVDSCommand, return: 896e24bd-5bd3-4961-b8c2-f06267a723e2, log id: 76d0f608 >2012-11-08 07:09:08,901 INFO [org.ovirt.engine.core.bll.EntityAsyncTask] (ajp-/127.0.0.1:8702-3) [3f92cb5c] EntityAsyncTask::Adding EntityMultiAsyncTasks object for entity 5cf389c5-51d0-46cc-9a5a-6042f734a901 >2012-11-08 07:09:08,901 INFO [org.ovirt.engine.core.bll.EntityMultiAsyncTasks] (ajp-/127.0.0.1:8702-3) [3f92cb5c] EntityMultiAsyncTasks::AttachTask: Attaching task a7803b0b-e5c1-413d-b016-1c2b2c01a8d8 to entity 5cf389c5-51d0-46cc-9a5a-6042f734a901. >2012-11-08 07:09:08,902 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskUtils] (ajp-/127.0.0.1:8702-3) [3f92cb5c] Adding task a7803b0b-e5c1-413d-b016-1c2b2c01a8d8 to DataBase >2012-11-08 07:09:08,934 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (ajp-/127.0.0.1:8702-3) [3f92cb5c] Adding task a7803b0b-e5c1-413d-b016-1c2b2c01a8d8 (Parent Command AddDisk, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters), polling hasn't started yet.. >2012-11-08 07:09:08,941 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (ajp-/127.0.0.1:8702-3) [3f92cb5c] BaseAsyncTask::StartPollingTask: Starting to poll task a7803b0b-e5c1-413d-b016-1c2b2c01a8d8. >2012-11-08 07:09:08,941 INFO [org.ovirt.engine.core.bll.EntityMultiAsyncTasks] (ajp-/127.0.0.1:8702-3) [3f92cb5c] EntityMultiAsyncTasks::StartPollingTask: Current Action Type for entity 5cf389c5-51d0-46cc-9a5a-6042f734a901 is AddDisk (determined by task a7803b0b-e5c1-413d-b016-1c2b2c01a8d8) >2012-11-08 07:09:17,845 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-57) Polling and updating Async Tasks: 3 tasks, 2 tasks to poll now >2012-11-08 07:09:17,893 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (QuartzScheduler_Worker-57) SPMAsyncTask::PollTask: Polling task a7803b0b-e5c1-413d-b016-1c2b2c01a8d8 (Parent Command AddDisk, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status running. >2012-11-08 07:09:17,893 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (QuartzScheduler_Worker-57) SPMAsyncTask::PollTask: Polling task a9c01093-106f-40d7-992c-e9d3bde0169c (Parent Command AddDisk, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status finished, result 'success'. >2012-11-08 07:09:17,896 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (QuartzScheduler_Worker-57) BaseAsyncTask::OnTaskEndSuccess: Task a9c01093-106f-40d7-992c-e9d3bde0169c (Parent Command AddDisk, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended successfully. >2012-11-08 07:09:17,897 INFO [org.ovirt.engine.core.bll.EntityAsyncTask] (QuartzScheduler_Worker-57) EntityAsyncTask::EndActionIfNecessary: All tasks of entity 7ca356eb-9853-4076-9347-172bccbf8d32 has ended -> executing EndAction >2012-11-08 07:09:17,897 INFO [org.ovirt.engine.core.bll.EntityAsyncTask] (QuartzScheduler_Worker-57) EntityAsyncTask::EndAction: Ending action for 1 tasks (entity ID: 7ca356eb-9853-4076-9347-172bccbf8d32): calling EndAction for action type AddDisk. >2012-11-08 07:09:17,897 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-57) Finished polling Tasks, will poll again in 10 seconds. >2012-11-08 07:09:17,897 INFO [org.ovirt.engine.core.bll.EntityAsyncTask] (pool-4-thread-50) EntityAsyncTask::EndCommandAction [within thread]context: Attempting to EndAction AddDisk >2012-11-08 07:09:17,903 INFO [org.ovirt.engine.core.bll.AddDiskCommand] (pool-4-thread-50) Ending command successfully: org.ovirt.engine.core.bll.AddDiskCommand >2012-11-08 07:09:17,903 INFO [org.ovirt.engine.core.bll.AddImageFromScratchCommand] (pool-4-thread-50) [4a7c974b] Ending command successfully: org.ovirt.engine.core.bll.AddImageFromScratchCommand >2012-11-08 07:09:17,905 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (pool-4-thread-50) [4a7c974b] START, GetImageInfoVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null, storageDomainId = a02e96be-cf4c-4d0f-8368-fd9d4aa659bf, imageGroupId = 7ca356eb-9853-4076-9347-172bccbf8d32, imageId = a6d0bc63-e895-451d-9662-b9a2e419891f), log id: 1aa4bf56 >2012-11-08 07:09:17,955 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (pool-4-thread-50) [4a7c974b] FINISH, GetImageInfoVDSCommand, return: org.ovirt.engine.core.common.businessentities.DiskImage@2a440a44, log id: 1aa4bf56 >2012-11-08 07:09:17,974 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UpdateVMVDSCommand] (pool-4-thread-50) START, UpdateVMVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null, storageDomainId = 00000000-0000-0000-0000-000000000000, infoDictionary.size = 1), log id: 18e793e2 >2012-11-08 07:09:18,007 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UpdateVMVDSCommand] (pool-4-thread-50) FINISH, UpdateVMVDSCommand, log id: 18e793e2 >2012-11-08 07:09:18,017 INFO [org.ovirt.engine.core.bll.EntityAsyncTask] (pool-4-thread-50) EntityAsyncTask::HandleEndActionResult [within thread]: EndAction for action type AddDisk completed, handling the result. >2012-11-08 07:09:18,017 INFO [org.ovirt.engine.core.bll.EntityAsyncTask] (pool-4-thread-50) EntityAsyncTask::HandleEndActionResult [within thread]: EndAction for action type AddDisk succeeded, clearing tasks. >2012-11-08 07:09:18,019 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (pool-4-thread-50) SPMAsyncTask::ClearAsyncTask: Attempting to clear task a9c01093-106f-40d7-992c-e9d3bde0169c >2012-11-08 07:09:18,019 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (pool-4-thread-50) START, SPMClearTaskVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null, taskId = a9c01093-106f-40d7-992c-e9d3bde0169c), log id: 1f8405c3 >2012-11-08 07:09:18,029 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (pool-4-thread-50) START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=a9c01093-106f-40d7-992c-e9d3bde0169c), log id: 5846b2ce >2012-11-08 07:09:18,064 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (pool-4-thread-50) FINISH, HSMClearTaskVDSCommand, log id: 5846b2ce >2012-11-08 07:09:18,064 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (pool-4-thread-50) FINISH, SPMClearTaskVDSCommand, log id: 1f8405c3 >2012-11-08 07:09:18,066 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (pool-4-thread-50) BaseAsyncTask::RemoveTaskFromDB: Removed task a9c01093-106f-40d7-992c-e9d3bde0169c from DataBase >2012-11-08 07:09:18,066 INFO [org.ovirt.engine.core.bll.EntityAsyncTask] (pool-4-thread-50) EntityAsyncTask::HandleEndActionResult [within thread]: Removing EntityMultiAsyncTasks object for entity 7ca356eb-9853-4076-9347-172bccbf8d32 >2012-11-08 07:09:19,399 INFO [org.ovirt.engine.core.bll.AddVmInterfaceCommand] (ajp-/127.0.0.1:8702-4) [1af40a86] Running command: AddVmInterfaceCommand internal: false. Entities affected : ID: 0ed98fdf-e2d5-4bf6-b94a-3cba410c4c8f Type: VM >2012-11-08 07:09:19,399 INFO [org.ovirt.engine.core.bll.MacPoolManager] (ajp-/127.0.0.1:8702-4) [1af40a86] MacPoolManager::allocateNewMac entered >2012-11-08 07:09:19,399 INFO [org.ovirt.engine.core.bll.MacPoolManager] (ajp-/127.0.0.1:8702-4) [1af40a86] MacPoolManager::allocateNewMac allocated mac = 00:1a:4a:46:22:65 >2012-11-08 07:09:19,417 INFO [org.ovirt.engine.core.bll.ActivateDeactivateVmNicCommand] (ajp-/127.0.0.1:8702-4) [675121a2] Running command: ActivateDeactivateVmNicCommand internal: true. Entities affected : ID: 0ed98fdf-e2d5-4bf6-b94a-3cba410c4c8f Type: VM >2012-11-08 07:09:56,297 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-26) Setting new tasks map. The map contains now 2 tasks >2012-11-08 07:09:58,010 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-33) Polling and updating Async Tasks: 2 tasks, 1 tasks to poll now >2012-11-08 07:09:58,051 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (QuartzScheduler_Worker-33) SPMAsyncTask::PollTask: Polling task a7803b0b-e5c1-413d-b016-1c2b2c01a8d8 (Parent Command AddDisk, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status finished, result 'success'. >2012-11-08 07:09:58,057 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (QuartzScheduler_Worker-33) BaseAsyncTask::OnTaskEndSuccess: Task a7803b0b-e5c1-413d-b016-1c2b2c01a8d8 (Parent Command AddDisk, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended successfully. >2012-11-08 07:09:58,057 INFO [org.ovirt.engine.core.bll.EntityAsyncTask] (QuartzScheduler_Worker-33) EntityAsyncTask::EndActionIfNecessary: All tasks of entity 5cf389c5-51d0-46cc-9a5a-6042f734a901 has ended -> executing EndAction >2012-11-08 07:09:58,057 INFO [org.ovirt.engine.core.bll.EntityAsyncTask] (QuartzScheduler_Worker-33) EntityAsyncTask::EndAction: Ending action for 1 tasks (entity ID: 5cf389c5-51d0-46cc-9a5a-6042f734a901): calling EndAction for action type AddDisk. >2012-11-08 07:09:58,057 INFO [org.ovirt.engine.core.bll.EntityAsyncTask] (pool-4-thread-47) EntityAsyncTask::EndCommandAction [within thread]context: Attempting to EndAction AddDisk >2012-11-08 07:09:58,065 INFO [org.ovirt.engine.core.bll.AddDiskCommand] (pool-4-thread-47) [5c108410] Ending command successfully: org.ovirt.engine.core.bll.AddDiskCommand >2012-11-08 07:09:58,065 INFO [org.ovirt.engine.core.bll.AddImageFromScratchCommand] (pool-4-thread-47) [3f92cb5c] Ending command successfully: org.ovirt.engine.core.bll.AddImageFromScratchCommand >2012-11-08 07:09:58,074 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (pool-4-thread-47) [3f92cb5c] START, GetImageInfoVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null, storageDomainId = e0cce407-ebea-4322-b47e-7aceb73a7a6f, imageGroupId = 5cf389c5-51d0-46cc-9a5a-6042f734a901, imageId = 896e24bd-5bd3-4961-b8c2-f06267a723e2), log id: 56368b8a >2012-11-08 07:09:58,132 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (pool-4-thread-47) [3f92cb5c] FINISH, GetImageInfoVDSCommand, return: org.ovirt.engine.core.common.businessentities.DiskImage@52c66f54, log id: 56368b8a >2012-11-08 07:09:58,152 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UpdateVMVDSCommand] (pool-4-thread-47) START, UpdateVMVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null, storageDomainId = 00000000-0000-0000-0000-000000000000, infoDictionary.size = 1), log id: 5a8a48bf >2012-11-08 07:09:58,185 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UpdateVMVDSCommand] (pool-4-thread-47) FINISH, UpdateVMVDSCommand, log id: 5a8a48bf >2012-11-08 07:09:58,198 INFO [org.ovirt.engine.core.bll.EntityAsyncTask] (pool-4-thread-47) EntityAsyncTask::HandleEndActionResult [within thread]: EndAction for action type AddDisk completed, handling the result. >2012-11-08 07:09:58,198 INFO [org.ovirt.engine.core.bll.EntityAsyncTask] (pool-4-thread-47) EntityAsyncTask::HandleEndActionResult [within thread]: EndAction for action type AddDisk succeeded, clearing tasks. >2012-11-08 07:09:58,201 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (pool-4-thread-47) SPMAsyncTask::ClearAsyncTask: Attempting to clear task a7803b0b-e5c1-413d-b016-1c2b2c01a8d8 >2012-11-08 07:09:58,201 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (pool-4-thread-47) START, SPMClearTaskVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null, taskId = a7803b0b-e5c1-413d-b016-1c2b2c01a8d8), log id: 754c01fe >2012-11-08 07:09:58,216 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (pool-4-thread-47) START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=a7803b0b-e5c1-413d-b016-1c2b2c01a8d8), log id: 78f29533 >2012-11-08 07:09:58,249 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (pool-4-thread-47) FINISH, HSMClearTaskVDSCommand, log id: 78f29533 >2012-11-08 07:09:58,249 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (pool-4-thread-47) FINISH, SPMClearTaskVDSCommand, log id: 754c01fe >2012-11-08 07:09:58,252 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (pool-4-thread-47) BaseAsyncTask::RemoveTaskFromDB: Removed task a7803b0b-e5c1-413d-b016-1c2b2c01a8d8 from DataBase >2012-11-08 07:09:58,252 INFO [org.ovirt.engine.core.bll.EntityAsyncTask] (pool-4-thread-47) EntityAsyncTask::HandleEndActionResult [within thread]: Removing EntityMultiAsyncTasks object for entity 5cf389c5-51d0-46cc-9a5a-6042f734a901 >2012-11-08 07:10:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-37) Autorecovering hosts is disabled, skipping >2012-11-08 07:10:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-37) Autorecovering storage domains is disabled, skipping >2012-11-08 07:10:26,297 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-79) Setting new tasks map. The map contains now 1 tasks >2012-11-08 07:10:56,790 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-10) [349db66c] START, IsValidVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 21658ae5 >2012-11-08 07:10:56,790 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-10) [349db66c] FINISH, IsValidVDSCommand, return: true, log id: 21658ae5 >2012-11-08 07:10:56,798 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp-/127.0.0.1:8702-10) [349db66c] START, IsVmDuringInitiatingVDSCommand( vmId = 67bd2d56-8e0b-4328-bcfd-ea7f4758c885), log id: 2ea18d3b >2012-11-08 07:10:56,798 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp-/127.0.0.1:8702-10) [349db66c] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 2ea18d3b >2012-11-08 07:10:56,838 INFO [org.ovirt.engine.core.bll.RunVmOnceCommand] (ajp-/127.0.0.1:8702-10) [349db66c] Running command: RunVmOnceCommand internal: false. Entities affected : ID: 67bd2d56-8e0b-4328-bcfd-ea7f4758c885 Type: VM, ID: 67bd2d56-8e0b-4328-bcfd-ea7f4758c885 Type: VM >2012-11-08 07:10:56,913 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-10) [349db66c] START, CreateVmVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=67bd2d56-8e0b-4328-bcfd-ea7f4758c885, vm=org.ovirt.engine.core.common.businessentities.VM@77b1650e), log id: 2590ebfd >2012-11-08 07:10:56,925 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8702-10) [349db66c] START, CreateVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=67bd2d56-8e0b-4328-bcfd-ea7f4758c885, vm=org.ovirt.engine.core.common.businessentities.VM@77b1650e), log id: 74c7e878 >2012-11-08 07:10:56,976 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8702-10) [349db66c] org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand spiceSslCipherSuite=DEFAULT,memSize=512,kvmEnable=true,smp=1,emulatedMachine=rhel6.3.0,vmType=kvm,keyboardLayout=en-us,nice=0,display=qxl,smpCoresPerSocket=1,spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard,timeOffset=-43200,transparentHugePages=true,vmId=67bd2d56-8e0b-4328-bcfd-ea7f4758c885,devices=[Ljava.util.Map;@78f36c9d,acpiEnable=true,vmName=b1,cpuType=Nehalem,custom={viodiskcache=none} >2012-11-08 07:10:56,976 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8702-10) [349db66c] FINISH, CreateVDSCommand, log id: 74c7e878 >2012-11-08 07:10:56,976 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-10) [349db66c] IncreasePendingVms::CreateVmIncreasing vds rhs-gp-srv11.lab.eng.blr.redhat.com pending vcpu count, now 1. Vm: b1 >2012-11-08 07:10:56,978 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-10) [349db66c] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 2590ebfd >2012-11-08 07:10:58,403 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (QuartzScheduler_Worker-49) START, DestroyVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=67bd2d56-8e0b-4328-bcfd-ea7f4758c885, force=false, secondsToWait=0, gracefully=false), log id: 4bc5f978 >2012-11-08 07:10:58,459 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (QuartzScheduler_Worker-49) FINISH, DestroyVDSCommand, log id: 4bc5f978 >2012-11-08 07:10:58,484 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-49) Running on vds during rerun failed vm: null >2012-11-08 07:10:58,484 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-49) vm b1 running in db and not running in vds - add to rerun treatment. vds rhs-gp-srv11.lab.eng.blr.redhat.com >2012-11-08 07:10:58,487 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-49) Rerun vm 67bd2d56-8e0b-4328-bcfd-ea7f4758c885. Called from vds rhs-gp-srv11.lab.eng.blr.redhat.com >2012-11-08 07:10:58,508 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsDynamicDataVDSCommand] (pool-4-thread-47) START, UpdateVdsDynamicDataVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vdsDynamic=org.ovirt.engine.core.common.businessentities.VdsDynamic@1944e16f), log id: 34756be0 >2012-11-08 07:10:58,509 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsDynamicDataVDSCommand] (pool-4-thread-47) FINISH, UpdateVdsDynamicDataVDSCommand, log id: 34756be0 >2012-11-08 07:10:58,533 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (pool-4-thread-47) START, IsValidVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 219bb1a >2012-11-08 07:10:58,534 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (pool-4-thread-47) FINISH, IsValidVDSCommand, return: true, log id: 219bb1a >2012-11-08 07:10:58,546 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (pool-4-thread-47) START, IsVmDuringInitiatingVDSCommand( vmId = 67bd2d56-8e0b-4328-bcfd-ea7f4758c885), log id: 7d484bcd >2012-11-08 07:10:58,546 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (pool-4-thread-47) FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 7d484bcd >2012-11-08 07:10:58,582 INFO [org.ovirt.engine.core.bll.VdsSelector] (pool-4-thread-47) VDS rhs-gp-srv11.lab.eng.blr.redhat.com 96739be4-2990-11e2-8577-52540035ed48 is the same host the VM is currently running on VDS rhs-client11.lab.eng.blr.redhat.com 90b581c2-2977-11e2-82cd-52540035ed48 is not in up status or belongs to the VM's cluster VDS rhs-gp-srv4.lab.eng.blr.redhat.com d2bd32d4-2992-11e2-ab59-52540035ed48 is not in up status or belongs to the VM's cluster >2012-11-08 07:10:58,582 WARN [org.ovirt.engine.core.bll.RunVmOnceCommand] (pool-4-thread-47) CanDoAction of action RunVmOnce failed. Reasons:VAR__ACTION__RUN,VAR__TYPE__VM,$VmName b1,VAR__ACTION__RUN,VAR__TYPE__VM,ACTION_TYPE_FAILED_VDS_VM_CLUSTER >2012-11-08 07:11:12,276 INFO [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (ajp-/127.0.0.1:8702-3) [23cc4f5e] Lock Acquired to object EngineLock [exclusiveLocks= key: b4 value: VM_NAME >, sharedLocks= ] >2012-11-08 07:11:12,284 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-3) [23cc4f5e] START, IsValidVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 15cdb186 >2012-11-08 07:11:12,285 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-3) [23cc4f5e] FINISH, IsValidVDSCommand, return: true, log id: 15cdb186 >2012-11-08 07:11:12,289 INFO [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (ajp-/127.0.0.1:8702-3) [23cc4f5e] Running command: AddVmFromScratchCommand internal: false. Entities affected : ID: 6cea47a6-2976-11e2-ad99-52540035ed48 Type: VdsGroups >2012-11-08 07:11:12,291 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-3) [23cc4f5e] START, IsValidVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 78e68dbe >2012-11-08 07:11:12,291 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-3) [23cc4f5e] FINISH, IsValidVDSCommand, return: true, log id: 78e68dbe >2012-11-08 07:11:12,301 INFO [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (ajp-/127.0.0.1:8702-3) [23cc4f5e] Lock freed to object EngineLock [exclusiveLocks= key: b4 value: VM_NAME >, sharedLocks= ] >2012-11-08 07:11:23,796 INFO [org.ovirt.engine.core.bll.AddDiskCommand] (ajp-/127.0.0.1:8702-7) [4a2831c4] Lock Acquired to object EngineLock [exclusiveLocks= key: e228f541-76b7-4a9b-b313-a6bacc2b73b9 value: VM_DISK_BOOT >, sharedLocks= key: e228f541-76b7-4a9b-b313-a6bacc2b73b9 value: VM >] >2012-11-08 07:11:23,801 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-7) [4a2831c4] START, IsValidVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 58482b53 >2012-11-08 07:11:23,802 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-7) [4a2831c4] FINISH, IsValidVDSCommand, return: true, log id: 58482b53 >2012-11-08 07:11:23,811 INFO [org.ovirt.engine.core.bll.AddDiskCommand] (ajp-/127.0.0.1:8702-7) [4a2831c4] Running command: AddDiskCommand internal: false. Entities affected : ID: e228f541-76b7-4a9b-b313-a6bacc2b73b9 Type: VM, ID: e0cce407-ebea-4322-b47e-7aceb73a7a6f Type: Storage >2012-11-08 07:11:23,837 INFO [org.ovirt.engine.core.bll.AddImageFromScratchCommand] (ajp-/127.0.0.1:8702-7) [4d0aa6f6] Running command: AddImageFromScratchCommand internal: true. Entities affected : ID: e0cce407-ebea-4322-b47e-7aceb73a7a6f Type: Storage >2012-11-08 07:11:23,846 INFO [org.ovirt.engine.core.bll.AddImageFromScratchCommand] (ajp-/127.0.0.1:8702-7) [4d0aa6f6] Lock freed to object EngineLock [exclusiveLocks= key: e228f541-76b7-4a9b-b313-a6bacc2b73b9 value: VM_DISK_BOOT >, sharedLocks= key: e228f541-76b7-4a9b-b313-a6bacc2b73b9 value: VM >] >2012-11-08 07:11:23,847 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (ajp-/127.0.0.1:8702-7) [4d0aa6f6] START, CreateImageVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = 3.1, storageDomainId = e0cce407-ebea-4322-b47e-7aceb73a7a6f, imageGroupId = e9e2a08f-d8df-4c31-92e3-3d920bb4ece3, imageSizeInBytes = 5368709120, volumeFormat = RAW, newImageId = aaa9497a-8ebb-4e50-aba5-ac720a8f79a8, newImageDescription = ), log id: 50dc1ad7 >2012-11-08 07:11:23,848 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (ajp-/127.0.0.1:8702-7) [4d0aa6f6] -- CreateImageVDSCommand::ExecuteIrsBrokerCommand: calling 'createVolume' with two new parameters: description and UUID >2012-11-08 07:11:23,848 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (ajp-/127.0.0.1:8702-7) [4d0aa6f6] -- createVolume parameters: > sdUUID=e0cce407-ebea-4322-b47e-7aceb73a7a6f > spUUID=3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 > imgGUID=e9e2a08f-d8df-4c31-92e3-3d920bb4ece3 > size=5,368,709,120 bytes > volFormat=RAW > volType=Preallocated > volUUID=aaa9497a-8ebb-4e50-aba5-ac720a8f79a8 > descr= > srcImgGUID=00000000-0000-0000-0000-000000000000 > srcVolUUID=00000000-0000-0000-0000-000000000000 > >2012-11-08 07:11:23,921 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (ajp-/127.0.0.1:8702-7) [4d0aa6f6] FINISH, CreateImageVDSCommand, return: aaa9497a-8ebb-4e50-aba5-ac720a8f79a8, log id: 50dc1ad7 >2012-11-08 07:11:23,923 INFO [org.ovirt.engine.core.bll.EntityAsyncTask] (ajp-/127.0.0.1:8702-7) [4d0aa6f6] EntityAsyncTask::Adding EntityMultiAsyncTasks object for entity e9e2a08f-d8df-4c31-92e3-3d920bb4ece3 >2012-11-08 07:11:23,923 INFO [org.ovirt.engine.core.bll.EntityMultiAsyncTasks] (ajp-/127.0.0.1:8702-7) [4d0aa6f6] EntityMultiAsyncTasks::AttachTask: Attaching task aa8d0763-b5ae-499e-b93f-1ec5491d7471 to entity e9e2a08f-d8df-4c31-92e3-3d920bb4ece3. >2012-11-08 07:11:23,925 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskUtils] (ajp-/127.0.0.1:8702-7) [4d0aa6f6] Adding task aa8d0763-b5ae-499e-b93f-1ec5491d7471 to DataBase >2012-11-08 07:11:23,939 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (ajp-/127.0.0.1:8702-7) [4d0aa6f6] Adding task aa8d0763-b5ae-499e-b93f-1ec5491d7471 (Parent Command AddDisk, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters), polling hasn't started yet.. >2012-11-08 07:11:23,947 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (ajp-/127.0.0.1:8702-7) [4d0aa6f6] BaseAsyncTask::StartPollingTask: Starting to poll task aa8d0763-b5ae-499e-b93f-1ec5491d7471. >2012-11-08 07:11:23,947 INFO [org.ovirt.engine.core.bll.EntityMultiAsyncTasks] (ajp-/127.0.0.1:8702-7) [4d0aa6f6] EntityMultiAsyncTasks::StartPollingTask: Current Action Type for entity e9e2a08f-d8df-4c31-92e3-3d920bb4ece3 is AddDisk (determined by task aa8d0763-b5ae-499e-b93f-1ec5491d7471) >2012-11-08 07:11:26,299 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-93) Setting new tasks map. The map contains now 1 tasks >2012-11-08 07:11:28,060 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-98) Polling and updating Async Tasks: 1 tasks, 1 tasks to poll now >2012-11-08 07:11:28,094 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (QuartzScheduler_Worker-98) SPMAsyncTask::PollTask: Polling task aa8d0763-b5ae-499e-b93f-1ec5491d7471 (Parent Command AddDisk, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status running. >2012-11-08 07:11:28,094 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-98) Finished polling Tasks, will poll again in 10 seconds. >2012-11-08 07:11:30,793 INFO [org.ovirt.engine.core.bll.AddVmInterfaceCommand] (ajp-/127.0.0.1:8702-14) [7e0bf10] Running command: AddVmInterfaceCommand internal: false. Entities affected : ID: e228f541-76b7-4a9b-b313-a6bacc2b73b9 Type: VM >2012-11-08 07:11:30,794 INFO [org.ovirt.engine.core.bll.MacPoolManager] (ajp-/127.0.0.1:8702-14) [7e0bf10] MacPoolManager::allocateNewMac entered >2012-11-08 07:11:30,794 INFO [org.ovirt.engine.core.bll.MacPoolManager] (ajp-/127.0.0.1:8702-14) [7e0bf10] MacPoolManager::allocateNewMac allocated mac = 00:1a:4a:46:22:66 >2012-11-08 07:11:30,842 INFO [org.ovirt.engine.core.bll.ActivateDeactivateVmNicCommand] (ajp-/127.0.0.1:8702-14) [3dfe8836] Running command: ActivateDeactivateVmNicCommand internal: true. Entities affected : ID: e228f541-76b7-4a9b-b313-a6bacc2b73b9 Type: VM >2012-11-08 07:12:08,239 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (QuartzScheduler_Worker-69) SPMAsyncTask::PollTask: Polling task aa8d0763-b5ae-499e-b93f-1ec5491d7471 (Parent Command AddDisk, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status finished, result 'success'. >2012-11-08 07:12:08,244 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (QuartzScheduler_Worker-69) BaseAsyncTask::OnTaskEndSuccess: Task aa8d0763-b5ae-499e-b93f-1ec5491d7471 (Parent Command AddDisk, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended successfully. >2012-11-08 07:12:08,245 INFO [org.ovirt.engine.core.bll.EntityAsyncTask] (QuartzScheduler_Worker-69) EntityAsyncTask::EndActionIfNecessary: All tasks of entity e9e2a08f-d8df-4c31-92e3-3d920bb4ece3 has ended -> executing EndAction >2012-11-08 07:12:08,248 INFO [org.ovirt.engine.core.bll.EntityAsyncTask] (QuartzScheduler_Worker-69) EntityAsyncTask::EndAction: Ending action for 1 tasks (entity ID: e9e2a08f-d8df-4c31-92e3-3d920bb4ece3): calling EndAction for action type AddDisk. >2012-11-08 07:12:08,249 INFO [org.ovirt.engine.core.bll.EntityAsyncTask] (pool-4-thread-50) EntityAsyncTask::EndCommandAction [within thread]context: Attempting to EndAction AddDisk >2012-11-08 07:12:08,256 INFO [org.ovirt.engine.core.bll.AddDiskCommand] (pool-4-thread-50) [4a2831c4] Ending command successfully: org.ovirt.engine.core.bll.AddDiskCommand >2012-11-08 07:12:08,257 INFO [org.ovirt.engine.core.bll.AddImageFromScratchCommand] (pool-4-thread-50) [4d0aa6f6] Ending command successfully: org.ovirt.engine.core.bll.AddImageFromScratchCommand >2012-11-08 07:12:08,267 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (pool-4-thread-50) [4d0aa6f6] START, GetImageInfoVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null, storageDomainId = e0cce407-ebea-4322-b47e-7aceb73a7a6f, imageGroupId = e9e2a08f-d8df-4c31-92e3-3d920bb4ece3, imageId = aaa9497a-8ebb-4e50-aba5-ac720a8f79a8), log id: 7726e3b7 >2012-11-08 07:12:08,318 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (pool-4-thread-50) [4d0aa6f6] FINISH, GetImageInfoVDSCommand, return: org.ovirt.engine.core.common.businessentities.DiskImage@a482aa6d, log id: 7726e3b7 >2012-11-08 07:12:08,336 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UpdateVMVDSCommand] (pool-4-thread-50) START, UpdateVMVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null, storageDomainId = 00000000-0000-0000-0000-000000000000, infoDictionary.size = 1), log id: 1239b1cf >2012-11-08 07:12:08,367 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UpdateVMVDSCommand] (pool-4-thread-50) FINISH, UpdateVMVDSCommand, log id: 1239b1cf >2012-11-08 07:12:08,375 INFO [org.ovirt.engine.core.bll.EntityAsyncTask] (pool-4-thread-50) EntityAsyncTask::HandleEndActionResult [within thread]: EndAction for action type AddDisk completed, handling the result. >2012-11-08 07:12:08,375 INFO [org.ovirt.engine.core.bll.EntityAsyncTask] (pool-4-thread-50) EntityAsyncTask::HandleEndActionResult [within thread]: EndAction for action type AddDisk succeeded, clearing tasks. >2012-11-08 07:12:08,376 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (pool-4-thread-50) SPMAsyncTask::ClearAsyncTask: Attempting to clear task aa8d0763-b5ae-499e-b93f-1ec5491d7471 >2012-11-08 07:12:08,377 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (pool-4-thread-50) START, SPMClearTaskVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null, taskId = aa8d0763-b5ae-499e-b93f-1ec5491d7471), log id: 4adc81f5 >2012-11-08 07:12:08,402 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (pool-4-thread-50) START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=aa8d0763-b5ae-499e-b93f-1ec5491d7471), log id: 781c9539 >2012-11-08 07:12:08,428 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (pool-4-thread-50) FINISH, HSMClearTaskVDSCommand, log id: 781c9539 >2012-11-08 07:12:08,428 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (pool-4-thread-50) FINISH, SPMClearTaskVDSCommand, log id: 4adc81f5 >2012-11-08 07:12:08,430 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (pool-4-thread-50) BaseAsyncTask::RemoveTaskFromDB: Removed task aa8d0763-b5ae-499e-b93f-1ec5491d7471 from DataBase >2012-11-08 07:12:08,430 INFO [org.ovirt.engine.core.bll.EntityAsyncTask] (pool-4-thread-50) EntityAsyncTask::HandleEndActionResult [within thread]: Removing EntityMultiAsyncTasks object for entity e9e2a08f-d8df-4c31-92e3-3d920bb4ece3 >2012-11-08 07:13:25,778 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (ajp-/127.0.0.1:8702-5) [7b61dda5] START, ValidateStorageServerConnectionVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: null, connection: guido.lab.eng.blr.redhat.com:/iso2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 71f425d1 >2012-11-08 07:13:25,796 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (ajp-/127.0.0.1:8702-5) [7b61dda5] FINISH, ValidateStorageServerConnectionVDSCommand, return: {00000000-0000-0000-0000-000000000000=0}, log id: 71f425d1 >2012-11-08 07:13:25,800 INFO [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-5) [7b61dda5] Running command: AddStorageServerConnectionCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2012-11-08 07:13:25,810 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-5) [7b61dda5] START, ConnectStorageServerVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 54270a93-f87f-477e-ad6d-e001aa34dae0, connection: guido.lab.eng.blr.redhat.com:/iso2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 2e3b4071 >2012-11-08 07:13:25,875 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-5) [7b61dda5] FINISH, ConnectStorageServerVDSCommand, return: {54270a93-f87f-477e-ad6d-e001aa34dae0=0}, log id: 2e3b4071 >2012-11-08 07:13:25,910 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-15) Failed to decrypt Data must not be longer than 256 bytes >2012-11-08 07:13:25,910 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (ajp-/127.0.0.1:8702-11) Failed to decrypt Data must not be longer than 256 bytes >2012-11-08 07:13:25,929 INFO [org.ovirt.engine.core.bll.storage.AddNFSStorageDomainCommand] (ajp-/127.0.0.1:8702-11) [7e5c5268] Running command: AddNFSStorageDomainCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2012-11-08 07:13:25,949 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-11) [7e5c5268] START, CreateStorageDomainVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storageDomain=org.ovirt.engine.core.common.businessentities.storage_domain_static@7a6d04bb, args=guido.lab.eng.blr.redhat.com:/iso2), log id: 47e0c4b2 >2012-11-08 07:13:26,299 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-18) Setting new tasks map. The map contains now 0 tasks >2012-11-08 07:13:26,299 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-18) Cleared all tasks of pool 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6. >2012-11-08 07:13:26,317 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-18) Could not find vds that is spm and non-operational. >2012-11-08 07:13:28,559 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-11) [7e5c5268] FINISH, CreateStorageDomainVDSCommand, log id: 47e0c4b2 >2012-11-08 07:13:28,570 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStorageDomainStatsVDSCommand] (ajp-/127.0.0.1:8702-11) [7e5c5268] START, GetStorageDomainStatsVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storageDomainId=7a268ac7-857a-4c53-966d-247deb509161), log id: 217ebe6e >2012-11-08 07:13:28,594 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStorageDomainStatsVDSCommand] (ajp-/127.0.0.1:8702-11) [7e5c5268] FINISH, GetStorageDomainStatsVDSCommand, return: org.ovirt.engine.core.common.businessentities.storage_domains@616b8ac1, log id: 217ebe6e >2012-11-08 07:13:28,666 INFO [org.ovirt.engine.core.bll.storage.RemoveStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-1) [6f8bb816] Running command: RemoveStorageServerConnectionCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2012-11-08 07:13:28,676 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-1) [6f8bb816] START, DisconnectStorageServerVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: null, connection: guido.lab.eng.blr.redhat.com:/iso2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 2250ffa8 >2012-11-08 07:13:30,884 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-1) [6f8bb816] FINISH, DisconnectStorageServerVDSCommand, return: {00000000-0000-0000-0000-000000000000=0}, log id: 2250ffa8 >2012-11-08 07:13:30,994 INFO [org.ovirt.engine.core.bll.storage.AttachStorageDomainToPoolCommand] (ajp-/127.0.0.1:8702-11) [56c4cfec] Running command: AttachStorageDomainToPoolCommand internal: false. Entities affected : ID: 7a268ac7-857a-4c53-966d-247deb509161 Type: Storage >2012-11-08 07:13:31,060 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (pool-4-thread-50) [755da319] START, ValidateStorageServerConnectionVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 658918ca-5dd4-4f39-9d39-f4a117794418, connection: guido.lab.eng.blr.redhat.com:/iso2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 42983f4a >2012-11-08 07:13:31,080 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (pool-4-thread-50) [755da319] FINISH, ValidateStorageServerConnectionVDSCommand, return: {658918ca-5dd4-4f39-9d39-f4a117794418=0}, log id: 42983f4a >2012-11-08 07:13:31,080 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (pool-4-thread-50) [755da319] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2012-11-08 07:13:31,094 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-4-thread-50) [755da319] START, ConnectStorageServerVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 658918ca-5dd4-4f39-9d39-f4a117794418, connection: guido.lab.eng.blr.redhat.com:/iso2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 13afd986 >2012-11-08 07:13:31,153 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-4-thread-50) [755da319] FINISH, ConnectStorageServerVDSCommand, return: {658918ca-5dd4-4f39-9d39-f4a117794418=0}, log id: 13afd986 >2012-11-08 07:13:31,154 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.AttachStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-11) [56c4cfec] START, AttachStorageDomainVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null, storageDomainId = 7a268ac7-857a-4c53-966d-247deb509161), log id: b92f619 >2012-11-08 07:13:34,389 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.AttachStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-11) [56c4cfec] FINISH, AttachStorageDomainVDSCommand, log id: b92f619 >2012-11-08 07:14:18,494 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-14) [444589b7] START, IsValidVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3e902f24 >2012-11-08 07:14:18,494 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-14) [444589b7] FINISH, IsValidVDSCommand, return: true, log id: 3e902f24 >2012-11-08 07:14:18,505 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp-/127.0.0.1:8702-14) [444589b7] START, IsVmDuringInitiatingVDSCommand( vmId = 8a0d2e26-440c-4d6c-91d3-1fc4d9b75990), log id: 163b348d >2012-11-08 07:14:18,505 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp-/127.0.0.1:8702-14) [444589b7] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 163b348d >2012-11-08 07:14:18,546 INFO [org.ovirt.engine.core.bll.RunVmOnceCommand] (ajp-/127.0.0.1:8702-14) [444589b7] Running command: RunVmOnceCommand internal: false. Entities affected : ID: 8a0d2e26-440c-4d6c-91d3-1fc4d9b75990 Type: VM >2012-11-08 07:14:18,620 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-14) [444589b7] START, CreateVmVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=8a0d2e26-440c-4d6c-91d3-1fc4d9b75990, vm=org.ovirt.engine.core.common.businessentities.VM@2544de6c), log id: 344f214c >2012-11-08 07:14:18,632 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8702-14) [444589b7] START, CreateVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=8a0d2e26-440c-4d6c-91d3-1fc4d9b75990, vm=org.ovirt.engine.core.common.businessentities.VM@2544de6c), log id: 5c872189 >2012-11-08 07:14:18,680 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8702-14) [444589b7] org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand spiceSslCipherSuite=DEFAULT,memSize=512,kvmEnable=true,smp=1,emulatedMachine=rhel6.3.0,vmType=kvm,keyboardLayout=en-us,nice=0,display=qxl,smpCoresPerSocket=1,spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard,timeOffset=-43200,transparentHugePages=true,vmId=8a0d2e26-440c-4d6c-91d3-1fc4d9b75990,devices=[Ljava.util.Map;@6fe42145,acpiEnable=true,vmName=disrep2,cpuType=Nehalem,custom={} >2012-11-08 07:14:18,681 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8702-14) [444589b7] FINISH, CreateVDSCommand, log id: 5c872189 >2012-11-08 07:14:18,681 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-14) [444589b7] IncreasePendingVms::CreateVmIncreasing vds rhs-gp-srv11.lab.eng.blr.redhat.com pending vcpu count, now 1. Vm: disrep2 >2012-11-08 07:14:18,683 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-14) [444589b7] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 344f214c >2012-11-08 07:14:18,852 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (QuartzScheduler_Worker-15) START, FullListVdsCommand( HostId = 96739be4-2990-11e2-8577-52540035ed48, vds=null, vmIds=[8a0d2e26-440c-4d6c-91d3-1fc4d9b75990]), log id: 21ace4b1 >2012-11-08 07:14:18,877 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (QuartzScheduler_Worker-15) FINISH, FullListVdsCommand, return: [Lorg.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcStruct;@5dda31bc, log id: 21ace4b1 >2012-11-08 07:14:18,883 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-15) Recieved a qxl Device without an address when processing VM 8a0d2e26-440c-4d6c-91d3-1fc4d9b75990 devices, skipping device: {specParams={vram=65536}, device=qxl, type=video, deviceId=6af57556-6fe7-4773-96d9-f7e2e01e06b0} >2012-11-08 07:14:18,883 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-15) Recieved a bridge Device without an address when processing VM 8a0d2e26-440c-4d6c-91d3-1fc4d9b75990 devices, skipping device: {bootOrder=1, nicModel=pv, specParams={}, device=bridge, macAddr=00:1a:4a:46:22:4f, type=interface, deviceId=fa95e907-b165-45b4-8c7b-c0a7cc8cdd36, filter=vdsm-no-mac-spoofing, network=rhevm} >2012-11-08 07:14:18,883 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-15) Recieved a memballoon Device without an address when processing VM 8a0d2e26-440c-4d6c-91d3-1fc4d9b75990 devices, skipping device: {specParams={model=virtio}, device=memballoon, type=balloon, deviceId=1f8a4aed-17e2-401d-97ef-52289b870342} >2012-11-08 07:14:18,883 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-15) Recieved a cdrom Device without an address when processing VM 8a0d2e26-440c-4d6c-91d3-1fc4d9b75990 devices, skipping device: {shared=false, iface=ide, index=2, specParams={path=}, device=cdrom, path=, type=disk, readonly=true, deviceId=7770db24-6638-4d4d-b096-86897477f32f} >2012-11-08 07:14:18,883 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-15) Recieved a disk Device without an address when processing VM 8a0d2e26-440c-4d6c-91d3-1fc4d9b75990 devices, skipping device: {shared=false, index=0, volumeID=d58cfacd-60a5-42df-a4fc-4f7c07a89eb5, propagateErrors=off, format=raw, type=disk, truesize=5368709120, reqsize=0, iface=virtio, bootOrder=2, volumeChain=[Ljava.lang.Object;@3b910280, imageID=8e861032-8233-4a00-9d20-49c3fb133088, domainID=e0cce407-ebea-4322-b47e-7aceb73a7a6f, specParams={}, optional=false, path=/rhev/data-center/3a17dcac-a6e3-4563-a4b0-f27d89b68dd6/e0cce407-ebea-4322-b47e-7aceb73a7a6f/images/8e861032-8233-4a00-9d20-49c3fb133088/d58cfacd-60a5-42df-a4fc-4f7c07a89eb5, device=disk, poolID=3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, readonly=false, deviceId=8e861032-8233-4a00-9d20-49c3fb133088, apparentsize=5368709120} >2012-11-08 07:14:20,971 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (QuartzScheduler_Worker-22) START, DestroyVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=8a0d2e26-440c-4d6c-91d3-1fc4d9b75990, force=false, secondsToWait=0, gracefully=false), log id: 65c470d0 >2012-11-08 07:14:21,032 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (QuartzScheduler_Worker-22) FINISH, DestroyVDSCommand, log id: 65c470d0 >2012-11-08 07:14:21,054 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-22) Running on vds during rerun failed vm: null >2012-11-08 07:14:21,055 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-22) vm disrep2 running in db and not running in vds - add to rerun treatment. vds rhs-gp-srv11.lab.eng.blr.redhat.com >2012-11-08 07:14:21,058 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-22) Rerun vm 8a0d2e26-440c-4d6c-91d3-1fc4d9b75990. Called from vds rhs-gp-srv11.lab.eng.blr.redhat.com >2012-11-08 07:14:21,084 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsDynamicDataVDSCommand] (pool-4-thread-50) START, UpdateVdsDynamicDataVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vdsDynamic=org.ovirt.engine.core.common.businessentities.VdsDynamic@1944e16f), log id: 472cc0a5 >2012-11-08 07:14:21,086 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsDynamicDataVDSCommand] (pool-4-thread-50) FINISH, UpdateVdsDynamicDataVDSCommand, log id: 472cc0a5 >2012-11-08 07:14:21,111 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (pool-4-thread-50) START, IsValidVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 36a6913c >2012-11-08 07:14:21,113 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (pool-4-thread-50) FINISH, IsValidVDSCommand, return: true, log id: 36a6913c >2012-11-08 07:14:21,129 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (pool-4-thread-50) START, IsVmDuringInitiatingVDSCommand( vmId = 8a0d2e26-440c-4d6c-91d3-1fc4d9b75990), log id: 61835206 >2012-11-08 07:14:21,129 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (pool-4-thread-50) FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 61835206 >2012-11-08 07:14:21,163 INFO [org.ovirt.engine.core.bll.VdsSelector] (pool-4-thread-50) VDS rhs-gp-srv11.lab.eng.blr.redhat.com 96739be4-2990-11e2-8577-52540035ed48 is the same host the VM is currently running on VDS rhs-client11.lab.eng.blr.redhat.com 90b581c2-2977-11e2-82cd-52540035ed48 is not in up status or belongs to the VM's cluster VDS rhs-gp-srv4.lab.eng.blr.redhat.com d2bd32d4-2992-11e2-ab59-52540035ed48 is not in up status or belongs to the VM's cluster >2012-11-08 07:14:21,163 WARN [org.ovirt.engine.core.bll.RunVmOnceCommand] (pool-4-thread-50) CanDoAction of action RunVmOnce failed. Reasons:VAR__ACTION__RUN,VAR__TYPE__VM,$VmName disrep2,VAR__ACTION__RUN,VAR__TYPE__VM,ACTION_TYPE_FAILED_VDS_VM_CLUSTER >2012-11-08 07:14:43,376 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp-/127.0.0.1:8702-13) Lock Acquired to object EngineLock [exclusiveLocks= key: 7a268ac7-857a-4c53-966d-247deb509161 value: STORAGE >, sharedLocks= ] >2012-11-08 07:14:43,420 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (pool-4-thread-47) [4575fe5d] Running command: ActivateStorageDomainCommand internal: false. Entities affected : ID: 7a268ac7-857a-4c53-966d-247deb509161 Type: Storage >2012-11-08 07:14:43,425 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (pool-4-thread-47) [4575fe5d] Lock freed to object EngineLock [exclusiveLocks= key: 7a268ac7-857a-4c53-966d-247deb509161 value: STORAGE >, sharedLocks= ] >2012-11-08 07:14:43,425 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (pool-4-thread-47) [4575fe5d] ActivateStorage Domain. Before Connect all hosts to pool. Time:11/8/12 7:14 AM >2012-11-08 07:14:43,470 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (pool-4-thread-50) [522a4f9] START, ValidateStorageServerConnectionVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 658918ca-5dd4-4f39-9d39-f4a117794418, connection: guido.lab.eng.blr.redhat.com:/iso2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 4b4526b4 >2012-11-08 07:14:43,490 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (pool-4-thread-50) [522a4f9] FINISH, ValidateStorageServerConnectionVDSCommand, return: {658918ca-5dd4-4f39-9d39-f4a117794418=0}, log id: 4b4526b4 >2012-11-08 07:14:43,490 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (pool-4-thread-50) [522a4f9] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2012-11-08 07:14:43,510 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-4-thread-50) [522a4f9] START, ConnectStorageServerVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 658918ca-5dd4-4f39-9d39-f4a117794418, connection: guido.lab.eng.blr.redhat.com:/iso2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 245f9d4f >2012-11-08 07:14:43,529 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-4-thread-50) [522a4f9] FINISH, ConnectStorageServerVDSCommand, return: {658918ca-5dd4-4f39-9d39-f4a117794418=0}, log id: 245f9d4f >2012-11-08 07:14:43,530 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (pool-4-thread-47) [4575fe5d] START, ActivateStorageDomainVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null, storageDomainId = 7a268ac7-857a-4c53-966d-247deb509161), log id: 1fc65a1d >2012-11-08 07:14:43,579 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (pool-4-thread-47) [4575fe5d] FINISH, ActivateStorageDomainVDSCommand, log id: 1fc65a1d >2012-11-08 07:14:43,579 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (pool-4-thread-47) [4575fe5d] ActivateStorage Domain. After Connect all hosts to pool. Time:11/8/12 7:14 AM >2012-11-08 07:14:43,640 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.RefreshStoragePoolVDSCommand] (pool-4-thread-50) [4575fe5d] START, RefreshStoragePoolVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, masterStorageDomainId=a02e96be-cf4c-4d0f-8368-fd9d4aa659bf, msterVersion=1), log id: 2dd7f795 >2012-11-08 07:14:46,367 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.RefreshStoragePoolVDSCommand] (pool-4-thread-50) [4575fe5d] FINISH, RefreshStoragePoolVDSCommand, log id: 2dd7f795 >2012-11-08 07:14:46,367 INFO [org.ovirt.engine.core.bll.storage.RefreshPoolSingleAsyncOperation] (pool-4-thread-50) [4575fe5d] Refreshed vds rhs-gp-srv11.lab.eng.blr.redhat.com in pool RHS-REBALANCE >2012-11-08 07:14:46,370 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (pool-4-thread-47) [4575fe5d] ActivateStorage Domain. After change storage pool status in vds. Time:11/8/12 7:14 AM >2012-11-08 07:14:46,372 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) [4575fe5d] START, GetIsoListVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6a4ece96 >2012-11-08 07:14:46,385 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) [4575fe5d] START, HsmGetIsoListVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId=3a17dcac-a6e3-4563-a4b0-f27d89b68dd6), log id: cf8f6a5 >2012-11-08 07:14:46,876 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) [4575fe5d] FINISH, HsmGetIsoListVDSCommand, return: [], log id: cf8f6a5 >2012-11-08 07:14:46,877 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) [4575fe5d] FINISH, GetIsoListVDSCommand, return: [], log id: 6a4ece96 >2012-11-08 07:14:46,892 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetFloppyListVDSCommand] (pool-4-thread-50) [4575fe5d] START, GetFloppyListVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 389f0949 >2012-11-08 07:14:46,994 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetFloppyListVDSCommand] (pool-4-thread-50) [4575fe5d] FINISH, GetFloppyListVDSCommand, return: [], log id: 389f0949 >2012-11-08 07:15:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-94) Autorecovering hosts is disabled, skipping >2012-11-08 07:15:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-94) Autorecovering storage domains is disabled, skipping >2012-11-08 07:15:12,743 INFO [org.ovirt.engine.core.bll.LoginAdminUserCommand] (ajp-/127.0.0.1:8702-14) Checking if user admin@internal is an admin, result true >2012-11-08 07:15:12,743 INFO [org.ovirt.engine.core.bll.LoginAdminUserCommand] (ajp-/127.0.0.1:8702-14) Running command: LoginAdminUserCommand internal: false. >2012-11-08 07:15:26,285 INFO [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (ajp-/127.0.0.1:8702-12) [554f43af] Lock Acquired to object EngineLock [exclusiveLocks= key: b5 value: VM_NAME >, sharedLocks= ] >2012-11-08 07:15:26,294 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-12) [554f43af] START, IsValidVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6da1899b >2012-11-08 07:15:26,295 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-12) [554f43af] FINISH, IsValidVDSCommand, return: true, log id: 6da1899b >2012-11-08 07:15:26,298 INFO [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (ajp-/127.0.0.1:8702-12) [554f43af] Running command: AddVmFromScratchCommand internal: false. Entities affected : ID: 6cea47a6-2976-11e2-ad99-52540035ed48 Type: VdsGroups >2012-11-08 07:15:26,299 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-12) [554f43af] START, IsValidVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 1699d17f >2012-11-08 07:15:26,299 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-12) [554f43af] FINISH, IsValidVDSCommand, return: true, log id: 1699d17f >2012-11-08 07:15:26,312 INFO [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (ajp-/127.0.0.1:8702-12) [554f43af] Lock freed to object EngineLock [exclusiveLocks= key: b5 value: VM_NAME >, sharedLocks= ] >2012-11-08 07:15:52,575 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-12) Checking if user admin@internal is an admin, result true >2012-11-08 07:15:52,575 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-12) Running command: LoginUserCommand internal: false. >2012-11-08 07:15:52,580 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-12) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2012-11-08 07:15:52,592 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-12) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2012-11-08 07:15:53,576 INFO [org.ovirt.engine.core.bll.AddDiskCommand] (ajp-/127.0.0.1:8702-1) [3e6f9ee4] Lock Acquired to object EngineLock [exclusiveLocks= key: 3d766d10-58ab-4306-b5c4-7d109c7b6c46 value: VM_DISK_BOOT >, sharedLocks= key: 3d766d10-58ab-4306-b5c4-7d109c7b6c46 value: VM >] >2012-11-08 07:15:53,579 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-1) [3e6f9ee4] START, IsValidVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 11648e1e >2012-11-08 07:15:53,580 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-1) [3e6f9ee4] FINISH, IsValidVDSCommand, return: true, log id: 11648e1e >2012-11-08 07:15:53,584 INFO [org.ovirt.engine.core.bll.AddDiskCommand] (ajp-/127.0.0.1:8702-1) [3e6f9ee4] Running command: AddDiskCommand internal: false. Entities affected : ID: 3d766d10-58ab-4306-b5c4-7d109c7b6c46 Type: VM, ID: a02e96be-cf4c-4d0f-8368-fd9d4aa659bf Type: Storage >2012-11-08 07:15:53,597 INFO [org.ovirt.engine.core.bll.AddImageFromScratchCommand] (ajp-/127.0.0.1:8702-1) [c8a2725] Running command: AddImageFromScratchCommand internal: true. Entities affected : ID: a02e96be-cf4c-4d0f-8368-fd9d4aa659bf Type: Storage >2012-11-08 07:15:53,603 INFO [org.ovirt.engine.core.bll.AddImageFromScratchCommand] (ajp-/127.0.0.1:8702-1) [c8a2725] Lock freed to object EngineLock [exclusiveLocks= key: 3d766d10-58ab-4306-b5c4-7d109c7b6c46 value: VM_DISK_BOOT >, sharedLocks= key: 3d766d10-58ab-4306-b5c4-7d109c7b6c46 value: VM >] >2012-11-08 07:15:53,604 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (ajp-/127.0.0.1:8702-1) [c8a2725] START, CreateImageVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = 3.1, storageDomainId = a02e96be-cf4c-4d0f-8368-fd9d4aa659bf, imageGroupId = f6d26f54-5fe4-4c74-8660-ae059dafd9a6, imageSizeInBytes = 5368709120, volumeFormat = RAW, newImageId = c5444624-5b51-4a11-a597-bb21e7285b85, newImageDescription = ), log id: 457551a0 >2012-11-08 07:15:53,604 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (ajp-/127.0.0.1:8702-1) [c8a2725] -- CreateImageVDSCommand::ExecuteIrsBrokerCommand: calling 'createVolume' with two new parameters: description and UUID >2012-11-08 07:15:53,604 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (ajp-/127.0.0.1:8702-1) [c8a2725] -- createVolume parameters: > sdUUID=a02e96be-cf4c-4d0f-8368-fd9d4aa659bf > spUUID=3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 > imgGUID=f6d26f54-5fe4-4c74-8660-ae059dafd9a6 > size=5,368,709,120 bytes > volFormat=RAW > volType=Preallocated > volUUID=c5444624-5b51-4a11-a597-bb21e7285b85 > descr= > srcImgGUID=00000000-0000-0000-0000-000000000000 > srcVolUUID=00000000-0000-0000-0000-000000000000 > >2012-11-08 07:15:53,673 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (ajp-/127.0.0.1:8702-1) [c8a2725] FINISH, CreateImageVDSCommand, return: c5444624-5b51-4a11-a597-bb21e7285b85, log id: 457551a0 >2012-11-08 07:15:53,675 INFO [org.ovirt.engine.core.bll.EntityAsyncTask] (ajp-/127.0.0.1:8702-1) [c8a2725] EntityAsyncTask::Adding EntityMultiAsyncTasks object for entity f6d26f54-5fe4-4c74-8660-ae059dafd9a6 >2012-11-08 07:15:53,675 INFO [org.ovirt.engine.core.bll.EntityMultiAsyncTasks] (ajp-/127.0.0.1:8702-1) [c8a2725] EntityMultiAsyncTasks::AttachTask: Attaching task 7eb12f23-dda0-4d80-bacd-2b758c4cefe7 to entity f6d26f54-5fe4-4c74-8660-ae059dafd9a6. >2012-11-08 07:15:53,676 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskUtils] (ajp-/127.0.0.1:8702-1) [c8a2725] Adding task 7eb12f23-dda0-4d80-bacd-2b758c4cefe7 to DataBase >2012-11-08 07:15:53,727 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (ajp-/127.0.0.1:8702-1) [c8a2725] Adding task 7eb12f23-dda0-4d80-bacd-2b758c4cefe7 (Parent Command AddDisk, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters), polling hasn't started yet.. >2012-11-08 07:15:53,734 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (ajp-/127.0.0.1:8702-1) [c8a2725] BaseAsyncTask::StartPollingTask: Starting to poll task 7eb12f23-dda0-4d80-bacd-2b758c4cefe7. >2012-11-08 07:15:53,734 INFO [org.ovirt.engine.core.bll.EntityMultiAsyncTasks] (ajp-/127.0.0.1:8702-1) [c8a2725] EntityMultiAsyncTasks::StartPollingTask: Current Action Type for entity f6d26f54-5fe4-4c74-8660-ae059dafd9a6 is AddDisk (determined by task 7eb12f23-dda0-4d80-bacd-2b758c4cefe7) >2012-11-08 07:15:58,253 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-100) Polling and updating Async Tasks: 1 tasks, 1 tasks to poll now >2012-11-08 07:15:58,289 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (QuartzScheduler_Worker-100) SPMAsyncTask::PollTask: Polling task 7eb12f23-dda0-4d80-bacd-2b758c4cefe7 (Parent Command AddDisk, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status running. >2012-11-08 07:15:58,289 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-100) Finished polling Tasks, will poll again in 10 seconds. >2012-11-08 07:16:01,258 INFO [org.ovirt.engine.core.bll.AddVmInterfaceCommand] (ajp-/127.0.0.1:8702-1) [6189ce99] Running command: AddVmInterfaceCommand internal: false. Entities affected : ID: 3d766d10-58ab-4306-b5c4-7d109c7b6c46 Type: VM >2012-11-08 07:16:01,259 INFO [org.ovirt.engine.core.bll.MacPoolManager] (ajp-/127.0.0.1:8702-1) [6189ce99] MacPoolManager::allocateNewMac entered >2012-11-08 07:16:01,259 INFO [org.ovirt.engine.core.bll.MacPoolManager] (ajp-/127.0.0.1:8702-1) [6189ce99] MacPoolManager::allocateNewMac allocated mac = 00:1a:4a:46:22:63 >2012-11-08 07:16:01,283 INFO [org.ovirt.engine.core.bll.ActivateDeactivateVmNicCommand] (ajp-/127.0.0.1:8702-1) [43ac8d35] Running command: ActivateDeactivateVmNicCommand internal: true. Entities affected : ID: 3d766d10-58ab-4306-b5c4-7d109c7b6c46 Type: VM >2012-11-08 07:16:15,861 INFO [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (ajp-/127.0.0.1:8702-7) [1d76de67] Lock Acquired to object EngineLock [exclusiveLocks= key: disrep4 value: VM_NAME >, sharedLocks= ] >2012-11-08 07:16:15,872 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-7) [1d76de67] START, IsValidVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 7ed3fc5 >2012-11-08 07:16:15,873 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-7) [1d76de67] FINISH, IsValidVDSCommand, return: true, log id: 7ed3fc5 >2012-11-08 07:16:15,876 INFO [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (ajp-/127.0.0.1:8702-7) [1d76de67] Running command: AddVmFromScratchCommand internal: false. Entities affected : ID: 6cea47a6-2976-11e2-ad99-52540035ed48 Type: VdsGroups >2012-11-08 07:16:15,876 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-7) [1d76de67] START, IsValidVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 48d5bd5d >2012-11-08 07:16:15,877 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-7) [1d76de67] FINISH, IsValidVDSCommand, return: true, log id: 48d5bd5d >2012-11-08 07:16:15,886 INFO [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (ajp-/127.0.0.1:8702-7) [1d76de67] Lock freed to object EngineLock [exclusiveLocks= key: disrep4 value: VM_NAME >, sharedLocks= ] >2012-11-08 07:16:25,909 INFO [org.ovirt.engine.core.bll.AddDiskCommand] (ajp-/127.0.0.1:8702-5) [5d4ed058] Lock Acquired to object EngineLock [exclusiveLocks= key: 28f91cc7-0841-425d-a0fd-8ea1fd15c636 value: VM_DISK_BOOT >, sharedLocks= key: 28f91cc7-0841-425d-a0fd-8ea1fd15c636 value: VM >] >2012-11-08 07:16:25,913 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-5) [5d4ed058] START, IsValidVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 353fb3eb >2012-11-08 07:16:25,913 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-5) [5d4ed058] FINISH, IsValidVDSCommand, return: true, log id: 353fb3eb >2012-11-08 07:16:25,935 INFO [org.ovirt.engine.core.bll.AddDiskCommand] (ajp-/127.0.0.1:8702-5) [5d4ed058] Running command: AddDiskCommand internal: false. Entities affected : ID: 28f91cc7-0841-425d-a0fd-8ea1fd15c636 Type: VM, ID: e0cce407-ebea-4322-b47e-7aceb73a7a6f Type: Storage >2012-11-08 07:16:26,025 INFO [org.ovirt.engine.core.bll.AddImageFromScratchCommand] (ajp-/127.0.0.1:8702-5) [21b701d9] Running command: AddImageFromScratchCommand internal: true. Entities affected : ID: e0cce407-ebea-4322-b47e-7aceb73a7a6f Type: Storage >2012-11-08 07:16:26,245 INFO [org.ovirt.engine.core.bll.AddImageFromScratchCommand] (ajp-/127.0.0.1:8702-5) [21b701d9] Lock freed to object EngineLock [exclusiveLocks= key: 28f91cc7-0841-425d-a0fd-8ea1fd15c636 value: VM_DISK_BOOT >, sharedLocks= key: 28f91cc7-0841-425d-a0fd-8ea1fd15c636 value: VM >] >2012-11-08 07:16:26,247 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (ajp-/127.0.0.1:8702-5) [21b701d9] START, CreateImageVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = 3.1, storageDomainId = e0cce407-ebea-4322-b47e-7aceb73a7a6f, imageGroupId = 95e2d4e4-adce-4d19-8a8c-9a1fed42c7b9, imageSizeInBytes = 5368709120, volumeFormat = RAW, newImageId = 4f394f68-499b-46b4-88b3-76b227bb7b7a, newImageDescription = ), log id: 55849daa >2012-11-08 07:16:26,247 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (ajp-/127.0.0.1:8702-5) [21b701d9] -- CreateImageVDSCommand::ExecuteIrsBrokerCommand: calling 'createVolume' with two new parameters: description and UUID >2012-11-08 07:16:26,247 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (ajp-/127.0.0.1:8702-5) [21b701d9] -- createVolume parameters: > sdUUID=e0cce407-ebea-4322-b47e-7aceb73a7a6f > spUUID=3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 > imgGUID=95e2d4e4-adce-4d19-8a8c-9a1fed42c7b9 > size=5,368,709,120 bytes > volFormat=RAW > volType=Preallocated > volUUID=4f394f68-499b-46b4-88b3-76b227bb7b7a > descr= > srcImgGUID=00000000-0000-0000-0000-000000000000 > srcVolUUID=00000000-0000-0000-0000-000000000000 > >2012-11-08 07:16:27,575 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (ajp-/127.0.0.1:8702-5) [21b701d9] FINISH, CreateImageVDSCommand, return: 4f394f68-499b-46b4-88b3-76b227bb7b7a, log id: 55849daa >2012-11-08 07:16:27,578 INFO [org.ovirt.engine.core.bll.EntityAsyncTask] (ajp-/127.0.0.1:8702-5) [21b701d9] EntityAsyncTask::Adding EntityMultiAsyncTasks object for entity 95e2d4e4-adce-4d19-8a8c-9a1fed42c7b9 >2012-11-08 07:16:27,578 INFO [org.ovirt.engine.core.bll.EntityMultiAsyncTasks] (ajp-/127.0.0.1:8702-5) [21b701d9] EntityMultiAsyncTasks::AttachTask: Attaching task bf37a116-9683-45f5-af02-5c0b65f96abf to entity 95e2d4e4-adce-4d19-8a8c-9a1fed42c7b9. >2012-11-08 07:16:27,579 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskUtils] (ajp-/127.0.0.1:8702-5) [21b701d9] Adding task bf37a116-9683-45f5-af02-5c0b65f96abf to DataBase >2012-11-08 07:16:27,606 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (ajp-/127.0.0.1:8702-5) [21b701d9] Adding task bf37a116-9683-45f5-af02-5c0b65f96abf (Parent Command AddDisk, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters), polling hasn't started yet.. >2012-11-08 07:16:27,615 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (ajp-/127.0.0.1:8702-5) [21b701d9] BaseAsyncTask::StartPollingTask: Starting to poll task bf37a116-9683-45f5-af02-5c0b65f96abf. >2012-11-08 07:16:27,615 INFO [org.ovirt.engine.core.bll.EntityMultiAsyncTasks] (ajp-/127.0.0.1:8702-5) [21b701d9] EntityMultiAsyncTasks::StartPollingTask: Current Action Type for entity 95e2d4e4-adce-4d19-8a8c-9a1fed42c7b9 is AddDisk (determined by task bf37a116-9683-45f5-af02-5c0b65f96abf) >2012-11-08 07:16:28,375 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-53) Polling and updating Async Tasks: 2 tasks, 2 tasks to poll now >2012-11-08 07:16:28,408 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (QuartzScheduler_Worker-53) SPMAsyncTask::PollTask: Polling task bf37a116-9683-45f5-af02-5c0b65f96abf (Parent Command AddDisk, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status running. >2012-11-08 07:16:28,408 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-53) Finished polling Tasks, will poll again in 10 seconds. >2012-11-08 07:16:31,305 INFO [org.ovirt.engine.core.bll.AddVmInterfaceCommand] (ajp-/127.0.0.1:8702-5) [52548863] Running command: AddVmInterfaceCommand internal: false. Entities affected : ID: 28f91cc7-0841-425d-a0fd-8ea1fd15c636 Type: VM >2012-11-08 07:16:31,306 INFO [org.ovirt.engine.core.bll.MacPoolManager] (ajp-/127.0.0.1:8702-5) [52548863] MacPoolManager::allocateNewMac entered >2012-11-08 07:16:31,306 INFO [org.ovirt.engine.core.bll.MacPoolManager] (ajp-/127.0.0.1:8702-5) [52548863] MacPoolManager::allocateNewMac allocated mac = 00:1a:4a:46:22:64 >2012-11-08 07:16:31,334 INFO [org.ovirt.engine.core.bll.ActivateDeactivateVmNicCommand] (ajp-/127.0.0.1:8702-5) [18243ed1] Running command: ActivateDeactivateVmNicCommand internal: true. Entities affected : ID: 28f91cc7-0841-425d-a0fd-8ea1fd15c636 Type: VM >2012-11-08 07:16:58,521 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (QuartzScheduler_Worker-17) [36467d8f] SPMAsyncTask::PollTask: Polling task 7eb12f23-dda0-4d80-bacd-2b758c4cefe7 (Parent Command AddDisk, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status finished, result 'success'. >2012-11-08 07:16:58,526 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (QuartzScheduler_Worker-17) [36467d8f] BaseAsyncTask::OnTaskEndSuccess: Task 7eb12f23-dda0-4d80-bacd-2b758c4cefe7 (Parent Command AddDisk, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended successfully. >2012-11-08 07:16:58,526 INFO [org.ovirt.engine.core.bll.EntityAsyncTask] (QuartzScheduler_Worker-17) [36467d8f] EntityAsyncTask::EndActionIfNecessary: All tasks of entity f6d26f54-5fe4-4c74-8660-ae059dafd9a6 has ended -> executing EndAction >2012-11-08 07:16:58,526 INFO [org.ovirt.engine.core.bll.EntityAsyncTask] (QuartzScheduler_Worker-17) [36467d8f] EntityAsyncTask::EndAction: Ending action for 1 tasks (entity ID: f6d26f54-5fe4-4c74-8660-ae059dafd9a6): calling EndAction for action type AddDisk. >2012-11-08 07:16:58,527 INFO [org.ovirt.engine.core.bll.EntityAsyncTask] (pool-4-thread-50) [36467d8f] EntityAsyncTask::EndCommandAction [within thread]context: Attempting to EndAction AddDisk >2012-11-08 07:16:58,534 INFO [org.ovirt.engine.core.bll.AddDiskCommand] (pool-4-thread-50) [3e6f9ee4] Ending command successfully: org.ovirt.engine.core.bll.AddDiskCommand >2012-11-08 07:16:58,534 INFO [org.ovirt.engine.core.bll.AddImageFromScratchCommand] (pool-4-thread-50) [c8a2725] Ending command successfully: org.ovirt.engine.core.bll.AddImageFromScratchCommand >2012-11-08 07:16:58,536 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (pool-4-thread-50) [c8a2725] START, GetImageInfoVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null, storageDomainId = a02e96be-cf4c-4d0f-8368-fd9d4aa659bf, imageGroupId = f6d26f54-5fe4-4c74-8660-ae059dafd9a6, imageId = c5444624-5b51-4a11-a597-bb21e7285b85), log id: 52ca0d86 >2012-11-08 07:16:58,583 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (pool-4-thread-50) [c8a2725] FINISH, GetImageInfoVDSCommand, return: org.ovirt.engine.core.common.businessentities.DiskImage@30493511, log id: 52ca0d86 >2012-11-08 07:16:58,903 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UpdateVMVDSCommand] (pool-4-thread-50) START, UpdateVMVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null, storageDomainId = 00000000-0000-0000-0000-000000000000, infoDictionary.size = 1), log id: 219455c8 >2012-11-08 07:16:58,946 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UpdateVMVDSCommand] (pool-4-thread-50) FINISH, UpdateVMVDSCommand, log id: 219455c8 >2012-11-08 07:16:58,953 INFO [org.ovirt.engine.core.bll.EntityAsyncTask] (pool-4-thread-50) EntityAsyncTask::HandleEndActionResult [within thread]: EndAction for action type AddDisk completed, handling the result. >2012-11-08 07:16:58,954 INFO [org.ovirt.engine.core.bll.EntityAsyncTask] (pool-4-thread-50) EntityAsyncTask::HandleEndActionResult [within thread]: EndAction for action type AddDisk succeeded, clearing tasks. >2012-11-08 07:16:58,955 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (pool-4-thread-50) SPMAsyncTask::ClearAsyncTask: Attempting to clear task 7eb12f23-dda0-4d80-bacd-2b758c4cefe7 >2012-11-08 07:16:58,955 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (pool-4-thread-50) START, SPMClearTaskVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null, taskId = 7eb12f23-dda0-4d80-bacd-2b758c4cefe7), log id: 78e07be2 >2012-11-08 07:16:58,966 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (pool-4-thread-50) START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=7eb12f23-dda0-4d80-bacd-2b758c4cefe7), log id: 5cdd9c22 >2012-11-08 07:16:59,004 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (pool-4-thread-50) FINISH, HSMClearTaskVDSCommand, log id: 5cdd9c22 >2012-11-08 07:16:59,004 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (pool-4-thread-50) FINISH, SPMClearTaskVDSCommand, log id: 78e07be2 >2012-11-08 07:16:59,006 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (pool-4-thread-50) BaseAsyncTask::RemoveTaskFromDB: Removed task 7eb12f23-dda0-4d80-bacd-2b758c4cefe7 from DataBase >2012-11-08 07:16:59,006 INFO [org.ovirt.engine.core.bll.EntityAsyncTask] (pool-4-thread-50) EntityAsyncTask::HandleEndActionResult [within thread]: Removing EntityMultiAsyncTasks object for entity f6d26f54-5fe4-4c74-8660-ae059dafd9a6 >2012-11-08 07:17:00,529 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-11) START, IsValidVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 1c0cb3da >2012-11-08 07:17:00,530 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-11) FINISH, IsValidVDSCommand, return: true, log id: 1c0cb3da >2012-11-08 07:17:00,530 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (ajp-/127.0.0.1:8702-11) START, GetIsoListVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 4c8327e3 >2012-11-08 07:17:00,541 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (ajp-/127.0.0.1:8702-11) START, HsmGetIsoListVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId=3a17dcac-a6e3-4563-a4b0-f27d89b68dd6), log id: 3353eb5d >2012-11-08 07:17:00,572 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (ajp-/127.0.0.1:8702-11) FINISH, HsmGetIsoListVDSCommand, return: [RHEL6.3-20120613.2-Server-x86_64-DVD1.iso], log id: 3353eb5d >2012-11-08 07:17:00,572 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (ajp-/127.0.0.1:8702-11) FINISH, GetIsoListVDSCommand, return: [RHEL6.3-20120613.2-Server-x86_64-DVD1.iso], log id: 4c8327e3 >2012-11-08 07:17:00,590 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-11) START, IsValidVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 5d610f0a >2012-11-08 07:17:00,591 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-11) FINISH, IsValidVDSCommand, return: true, log id: 5d610f0a >2012-11-08 07:17:00,591 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetFloppyListVDSCommand] (ajp-/127.0.0.1:8702-11) START, GetFloppyListVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6f86fe93 >2012-11-08 07:17:00,620 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetFloppyListVDSCommand] (ajp-/127.0.0.1:8702-11) FINISH, GetFloppyListVDSCommand, return: [], log id: 6f86fe93 >2012-11-08 07:17:18,598 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (QuartzScheduler_Worker-47) SPMAsyncTask::PollTask: Polling task bf37a116-9683-45f5-af02-5c0b65f96abf (Parent Command AddDisk, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status finished, result 'success'. >2012-11-08 07:17:18,602 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (QuartzScheduler_Worker-47) BaseAsyncTask::OnTaskEndSuccess: Task bf37a116-9683-45f5-af02-5c0b65f96abf (Parent Command AddDisk, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended successfully. >2012-11-08 07:17:18,602 INFO [org.ovirt.engine.core.bll.EntityAsyncTask] (QuartzScheduler_Worker-47) EntityAsyncTask::EndActionIfNecessary: All tasks of entity 95e2d4e4-adce-4d19-8a8c-9a1fed42c7b9 has ended -> executing EndAction >2012-11-08 07:17:18,602 INFO [org.ovirt.engine.core.bll.EntityAsyncTask] (QuartzScheduler_Worker-47) EntityAsyncTask::EndAction: Ending action for 1 tasks (entity ID: 95e2d4e4-adce-4d19-8a8c-9a1fed42c7b9): calling EndAction for action type AddDisk. >2012-11-08 07:17:18,602 INFO [org.ovirt.engine.core.bll.EntityAsyncTask] (pool-4-thread-50) EntityAsyncTask::EndCommandAction [within thread]context: Attempting to EndAction AddDisk >2012-11-08 07:17:18,611 INFO [org.ovirt.engine.core.bll.AddDiskCommand] (pool-4-thread-50) [5d4ed058] Ending command successfully: org.ovirt.engine.core.bll.AddDiskCommand >2012-11-08 07:17:18,611 INFO [org.ovirt.engine.core.bll.AddImageFromScratchCommand] (pool-4-thread-50) [21b701d9] Ending command successfully: org.ovirt.engine.core.bll.AddImageFromScratchCommand >2012-11-08 07:17:18,614 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (pool-4-thread-50) [21b701d9] START, GetImageInfoVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null, storageDomainId = e0cce407-ebea-4322-b47e-7aceb73a7a6f, imageGroupId = 95e2d4e4-adce-4d19-8a8c-9a1fed42c7b9, imageId = 4f394f68-499b-46b4-88b3-76b227bb7b7a), log id: 6f9bab63 >2012-11-08 07:17:18,671 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (pool-4-thread-50) [21b701d9] FINISH, GetImageInfoVDSCommand, return: org.ovirt.engine.core.common.businessentities.DiskImage@9524841d, log id: 6f9bab63 >2012-11-08 07:17:18,684 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UpdateVMVDSCommand] (pool-4-thread-50) START, UpdateVMVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null, storageDomainId = 00000000-0000-0000-0000-000000000000, infoDictionary.size = 1), log id: 45e01c89 >2012-11-08 07:17:18,716 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UpdateVMVDSCommand] (pool-4-thread-50) FINISH, UpdateVMVDSCommand, log id: 45e01c89 >2012-11-08 07:17:18,725 INFO [org.ovirt.engine.core.bll.EntityAsyncTask] (pool-4-thread-50) EntityAsyncTask::HandleEndActionResult [within thread]: EndAction for action type AddDisk completed, handling the result. >2012-11-08 07:17:18,725 INFO [org.ovirt.engine.core.bll.EntityAsyncTask] (pool-4-thread-50) EntityAsyncTask::HandleEndActionResult [within thread]: EndAction for action type AddDisk succeeded, clearing tasks. >2012-11-08 07:17:18,726 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (pool-4-thread-50) SPMAsyncTask::ClearAsyncTask: Attempting to clear task bf37a116-9683-45f5-af02-5c0b65f96abf >2012-11-08 07:17:18,726 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (pool-4-thread-50) START, SPMClearTaskVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null, taskId = bf37a116-9683-45f5-af02-5c0b65f96abf), log id: 4ea05df9 >2012-11-08 07:17:18,736 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (pool-4-thread-50) START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=bf37a116-9683-45f5-af02-5c0b65f96abf), log id: 69d3f7ac >2012-11-08 07:17:18,761 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (pool-4-thread-50) FINISH, HSMClearTaskVDSCommand, log id: 69d3f7ac >2012-11-08 07:17:18,761 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (pool-4-thread-50) FINISH, SPMClearTaskVDSCommand, log id: 4ea05df9 >2012-11-08 07:17:18,763 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (pool-4-thread-50) BaseAsyncTask::RemoveTaskFromDB: Removed task bf37a116-9683-45f5-af02-5c0b65f96abf from DataBase >2012-11-08 07:17:18,763 INFO [org.ovirt.engine.core.bll.EntityAsyncTask] (pool-4-thread-50) EntityAsyncTask::HandleEndActionResult [within thread]: Removing EntityMultiAsyncTasks object for entity 95e2d4e4-adce-4d19-8a8c-9a1fed42c7b9 >2012-11-08 07:18:26,319 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-76) Setting new tasks map. The map contains now 0 tasks >2012-11-08 07:18:26,319 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-76) Cleared all tasks of pool 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6. >2012-11-08 07:18:26,334 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-76) Could not find vds that is spm and non-operational. >2012-11-08 07:19:00,822 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Checking if user admin@internal is an admin, result true >2012-11-08 07:19:00,823 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2012-11-08 07:19:00,833 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2012-11-08 07:19:00,833 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2012-11-08 07:19:23,828 INFO [org.ovirt.engine.core.bll.LoginAdminUserCommand] (ajp-/127.0.0.1:8702-12) Checking if user admin@internal is an admin, result true >2012-11-08 07:19:23,828 INFO [org.ovirt.engine.core.bll.LoginAdminUserCommand] (ajp-/127.0.0.1:8702-12) Running command: LoginAdminUserCommand internal: false. >2012-11-08 07:19:55,939 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-36) DbUserCacheManager::refreshAllUserData() - entered >2012-11-08 07:20:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-38) Autorecovering hosts is disabled, skipping >2012-11-08 07:20:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-38) Autorecovering storage domains is disabled, skipping >2012-11-08 07:22:09,830 INFO [org.ovirt.engine.core.bll.LoginAdminUserCommand] (ajp-/127.0.0.1:8702-2) Checking if user admin@internal is an admin, result true >2012-11-08 07:22:09,831 INFO [org.ovirt.engine.core.bll.LoginAdminUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginAdminUserCommand internal: false. >2012-11-08 07:22:34,698 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-7) [1d8855ac] START, IsValidVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 2cd47088 >2012-11-08 07:22:34,699 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-7) [1d8855ac] FINISH, IsValidVDSCommand, return: true, log id: 2cd47088 >2012-11-08 07:22:34,713 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp-/127.0.0.1:8702-7) [1d8855ac] START, IsVmDuringInitiatingVDSCommand( vmId = 67bd2d56-8e0b-4328-bcfd-ea7f4758c885), log id: 4abfa069 >2012-11-08 07:22:34,713 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp-/127.0.0.1:8702-7) [1d8855ac] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 4abfa069 >2012-11-08 07:22:34,753 INFO [org.ovirt.engine.core.bll.RunVmOnceCommand] (ajp-/127.0.0.1:8702-7) [1d8855ac] Running command: RunVmOnceCommand internal: false. Entities affected : ID: 67bd2d56-8e0b-4328-bcfd-ea7f4758c885 Type: VM, ID: 67bd2d56-8e0b-4328-bcfd-ea7f4758c885 Type: VM >2012-11-08 07:22:34,813 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsoPrefixVDSCommand] (ajp-/127.0.0.1:8702-7) [1d8855ac] START, IsoPrefixVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 39b01b7e >2012-11-08 07:22:34,843 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsoPrefixVDSCommand] (ajp-/127.0.0.1:8702-7) [1d8855ac] FINISH, IsoPrefixVDSCommand, return: /rhev/data-center/3a17dcac-a6e3-4563-a4b0-f27d89b68dd6/7a268ac7-857a-4c53-966d-247deb509161/images/11111111-1111-1111-1111-111111111111, log id: 39b01b7e >2012-11-08 07:22:34,860 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-7) [1d8855ac] START, CreateVmVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=67bd2d56-8e0b-4328-bcfd-ea7f4758c885, vm=org.ovirt.engine.core.common.businessentities.VM@8eefeb6), log id: 58dd0ac8 >2012-11-08 07:22:34,872 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8702-7) [1d8855ac] START, CreateVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=67bd2d56-8e0b-4328-bcfd-ea7f4758c885, vm=org.ovirt.engine.core.common.businessentities.VM@8eefeb6), log id: 72345d4b >2012-11-08 07:22:34,922 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8702-7) [1d8855ac] org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand spiceSslCipherSuite=DEFAULT,memSize=512,kvmEnable=true,smp=1,emulatedMachine=rhel6.3.0,vmType=kvm,keyboardLayout=en-us,nice=0,display=qxl,smpCoresPerSocket=1,spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard,timeOffset=-43200,transparentHugePages=true,vmId=67bd2d56-8e0b-4328-bcfd-ea7f4758c885,devices=[Ljava.util.Map;@6c1c323e,acpiEnable=true,vmName=b1,cpuType=Nehalem,custom={viodiskcache=none} >2012-11-08 07:22:34,922 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8702-7) [1d8855ac] FINISH, CreateVDSCommand, log id: 72345d4b >2012-11-08 07:22:34,922 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-7) [1d8855ac] IncreasePendingVms::CreateVmIncreasing vds rhs-gp-srv11.lab.eng.blr.redhat.com pending vcpu count, now 1. Vm: b1 >2012-11-08 07:22:34,928 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-7) [1d8855ac] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 58dd0ac8 >2012-11-08 07:22:37,819 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (QuartzScheduler_Worker-31) [6143973] START, DestroyVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=67bd2d56-8e0b-4328-bcfd-ea7f4758c885, force=false, secondsToWait=0, gracefully=false), log id: 5bb99436 >2012-11-08 07:22:37,876 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (QuartzScheduler_Worker-31) [6143973] FINISH, DestroyVDSCommand, log id: 5bb99436 >2012-11-08 07:22:37,892 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-31) [6143973] Running on vds during rerun failed vm: null >2012-11-08 07:22:37,893 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-31) [6143973] vm b1 running in db and not running in vds - add to rerun treatment. vds rhs-gp-srv11.lab.eng.blr.redhat.com >2012-11-08 07:22:37,898 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-31) [6143973] Rerun vm 67bd2d56-8e0b-4328-bcfd-ea7f4758c885. Called from vds rhs-gp-srv11.lab.eng.blr.redhat.com >2012-11-08 07:22:37,923 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsDynamicDataVDSCommand] (pool-4-thread-47) [6143973] START, UpdateVdsDynamicDataVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vdsDynamic=org.ovirt.engine.core.common.businessentities.VdsDynamic@dbea1aad), log id: 1b5c7182 >2012-11-08 07:22:37,925 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsDynamicDataVDSCommand] (pool-4-thread-47) [6143973] FINISH, UpdateVdsDynamicDataVDSCommand, log id: 1b5c7182 >2012-11-08 07:22:37,954 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (pool-4-thread-47) [6143973] START, IsValidVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 2a51c49a >2012-11-08 07:22:37,955 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (pool-4-thread-47) [6143973] FINISH, IsValidVDSCommand, return: true, log id: 2a51c49a >2012-11-08 07:22:37,972 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (pool-4-thread-47) [6143973] START, IsVmDuringInitiatingVDSCommand( vmId = 67bd2d56-8e0b-4328-bcfd-ea7f4758c885), log id: 37a6d492 >2012-11-08 07:22:37,972 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (pool-4-thread-47) [6143973] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 37a6d492 >2012-11-08 07:22:38,014 INFO [org.ovirt.engine.core.bll.VdsSelector] (pool-4-thread-47) [6143973] VDS rhs-gp-srv11.lab.eng.blr.redhat.com 96739be4-2990-11e2-8577-52540035ed48 is the same host the VM is currently running on VDS rhs-client11.lab.eng.blr.redhat.com 90b581c2-2977-11e2-82cd-52540035ed48 is not in up status or belongs to the VM's cluster VDS rhs-gp-srv4.lab.eng.blr.redhat.com d2bd32d4-2992-11e2-ab59-52540035ed48 is not in up status or belongs to the VM's cluster >2012-11-08 07:22:38,014 WARN [org.ovirt.engine.core.bll.RunVmOnceCommand] (pool-4-thread-47) [6143973] CanDoAction of action RunVmOnce failed. Reasons:VAR__ACTION__RUN,VAR__TYPE__VM,$VmName b1,VAR__ACTION__RUN,VAR__TYPE__VM,ACTION_TYPE_FAILED_VDS_VM_CLUSTER >2012-11-08 07:25:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-1) Autorecovering hosts is disabled, skipping >2012-11-08 07:25:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-1) Autorecovering storage domains is disabled, skipping >2012-11-08 07:29:16,055 INFO [org.ovirt.engine.core.bll.RegisterVdsQuery] (ajp-/127.0.0.1:8702-1) Running Command: RegisterVds >2012-11-08 07:29:16,057 WARN [org.ovirt.engine.core.dal.dbbroker.generic.DBConfigUtils] (ajp-/127.0.0.1:8702-1) Adding new value to configuration cache. >2012-11-08 07:29:16,057 WARN [org.ovirt.engine.core.dal.dbbroker.generic.DBConfigUtils] (ajp-/127.0.0.1:8702-1) Didnt find the value of LogVdsRegistration in DB for version general - using default: false >2012-11-08 07:29:16,159 INFO [org.ovirt.engine.core.bll.AddVdsCommand] (pool-4-thread-48) [24cec154] Running command: AddVdsCommand internal: true. Entities affected : ID: 99408929-82cf-4dc7-a532-9d998063fa95 Type: VdsGroups >2012-11-08 07:29:16,169 INFO [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (pool-4-thread-48) [42b0fbf7] Running command: AddVdsSpmIdCommand internal: true. Entities affected : ID: e9163546-299f-11e2-b43c-52540035ed48 Type: VDS >2012-11-08 07:29:16,173 INFO [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (pool-4-thread-48) [42b0fbf7] Lock freed to object EngineLock [exclusiveLocks= key: 928c7030-2974-11e2-9380-52540035ed48 value: REGISTER_VDS >, sharedLocks= ] >2012-11-08 07:29:16,173 ERROR [org.ovirt.engine.core.vdsbroker.ResourceManager] (pool-4-thread-48) [42b0fbf7] Cannot get vdsManager for vdsid=e9163546-299f-11e2-b43c-52540035ed48 >2012-11-08 07:29:16,174 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (pool-4-thread-48) [42b0fbf7] START, RemoveVdsVDSCommand(HostName = rhs-arch-srv2.lab.eng.blr.redhat.com, HostId = e9163546-299f-11e2-b43c-52540035ed48), log id: 59634e57 >2012-11-08 07:29:16,174 ERROR [org.ovirt.engine.core.vdsbroker.ResourceManager] (pool-4-thread-48) [42b0fbf7] Cannot get vdsManager for vdsid=e9163546-299f-11e2-b43c-52540035ed48 >2012-11-08 07:29:16,174 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (pool-4-thread-48) [42b0fbf7] FINISH, RemoveVdsVDSCommand, log id: 59634e57 >2012-11-08 07:29:16,174 ERROR [org.ovirt.engine.core.vdsbroker.ResourceManager] (pool-4-thread-48) [42b0fbf7] Cannot get vdsManager for vdsid=e9163546-299f-11e2-b43c-52540035ed48 >2012-11-08 07:29:16,175 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (pool-4-thread-48) [42b0fbf7] START, AddVdsVDSCommand(HostName = rhs-arch-srv2.lab.eng.blr.redhat.com, HostId = e9163546-299f-11e2-b43c-52540035ed48), log id: 14795cb1 >2012-11-08 07:29:16,175 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (pool-4-thread-48) [42b0fbf7] AddVds - entered , starting logic to add VDS e9163546-299f-11e2-b43c-52540035ed48 >2012-11-08 07:29:16,176 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (pool-4-thread-48) [42b0fbf7] AddVds - VDS e9163546-299f-11e2-b43c-52540035ed48 was added, will try to add it to the resource manager >2012-11-08 07:29:16,177 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (pool-4-thread-48) [42b0fbf7] Eneterd VdsManager:constructor >2012-11-08 07:29:16,177 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (pool-4-thread-48) [42b0fbf7] vdsBroker(10.70.36.2,54,321) >2012-11-08 07:29:16,177 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (pool-4-thread-48) [42b0fbf7] ResourceManager::AddVds - VDS e9163546-299f-11e2-b43c-52540035ed48 was added to the Resource Manager >2012-11-08 07:29:16,177 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (pool-4-thread-48) [42b0fbf7] FINISH, AddVdsVDSCommand, log id: 14795cb1 >2012-11-08 07:29:16,184 INFO [org.ovirt.engine.core.bll.RegisterVdsQuery] (ajp-/127.0.0.1:8702-1) RegisterVdsQuery::Register - Registered a new VDS pending approval - Name: rhs-arch-srv2.lab.eng.blr.redhat.com, Hostname: 10.70.36.2, UniqueID: 00000000-0000-0000-0000-0025902F9398 >2012-11-08 07:29:16,188 INFO [org.ovirt.engine.core.register.RegisterServlet] (ajp-/127.0.0.1:8702-1) Succeeded to run RegisterVds. >2012-11-08 07:30:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-75) Autorecovering hosts is disabled, skipping >2012-11-08 07:30:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-75) Autorecovering storage domains is disabled, skipping >2012-11-08 07:35:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-75) Autorecovering hosts is disabled, skipping >2012-11-08 07:35:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-75) Autorecovering storage domains is disabled, skipping >2012-11-08 07:36:24,210 ERROR [org.ovirt.engine.core.bll.GetoVirtISOsQuery] (ajp-/127.0.0.1:8702-13) ovirt ISOs directory not found. Search in: /usr/share/rhev-hypervisor >2012-11-08 07:36:33,941 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp-/127.0.0.1:8702-8) [5a3b8e83] Running command: UpdateVdsCommand internal: false. Entities affected : ID: e9163546-299f-11e2-b43c-52540035ed48 Type: VDS >2012-11-08 07:36:34,057 INFO [org.ovirt.engine.core.bll.ApproveVdsCommand] (pool-4-thread-48) [2cb216e8] Running command: ApproveVdsCommand internal: false. Entities affected : ID: e9163546-299f-11e2-b43c-52540035ed48 Type: VDS >2012-11-08 07:36:34,057 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-4-thread-48) [2cb216e8] Before Installation pool-4-thread-48, Powerclient/oVirtNode case: setting status to installing >2012-11-08 07:36:34,058 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-48) [2cb216e8] START, SetVdsStatusVDSCommand(HostName = rhs-arch-srv2.lab.eng.blr.redhat.com, HostId = e9163546-299f-11e2-b43c-52540035ed48, status=Installing, nonOperationalReason=NONE), log id: 3947988e >2012-11-08 07:36:34,063 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-48) [2cb216e8] FINISH, SetVdsStatusVDSCommand, log id: 3947988e >2012-11-08 07:36:34,065 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-4-thread-48) [2cb216e8] Before Installation pool-4-thread-48 >2012-11-08 07:36:34,065 INFO [org.ovirt.engine.core.bll.OVirtInstaller] (pool-4-thread-48) [2cb216e8] Installation of 10.70.36.2. Executing oVirt installation stage. (Stage: Starting Host installation) >2012-11-08 07:36:34,065 INFO [org.ovirt.engine.core.bll.OVirtInstaller] (pool-4-thread-48) [2cb216e8] Installation of 10.70.36.2. Executing oVirt installation stage. (Stage: Connecting to Host) >2012-11-08 07:36:34,163 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-48) [2cb216e8] Installation of 10.70.36.2. Received message: <BSTRAP component='RHEV_INSTALL' status='OK' message='Connected to Host 10.70.36.2 with SSH key fingerprint: d3:61:44:22:06:7a:76:2a:e3:12:7e:8e:41:56:25:a0'/>. FYI. (Stage: Connecting to Host) >2012-11-08 07:36:34,196 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-48) [2cb216e8] Installation of 10.70.36.2. Successfully connected to server ssh. (Stage: Connecting to Host) >2012-11-08 07:36:34,196 INFO [org.ovirt.engine.core.bll.OVirtInstaller] (pool-4-thread-48) [2cb216e8] Installation of 10.70.36.2. Executing oVirt installation stage. (Stage: Running first installation script on Host) >2012-11-08 07:36:34,196 INFO [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-4-thread-48) SSH execute 10.70.36.2 '/usr/share/vdsm-reg/vdsm-gen-cert -O "lab.eng.blr.redhat.com" 10.70.36.2 45ec7327-0abd-4564-9c77-f8c036cb475a' >2012-11-08 07:36:34,578 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-95) Initializing Host: rhs-arch-srv2.lab.eng.blr.redhat.com >2012-11-08 07:36:36,027 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-43) Installation of 10.70.36.2. Received message: <BSTRAP component='Encryption setup' status='OK'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 07:36:36,033 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-43) Installation of 10.70.36.2. Received message: <BSTRAP component='RHEV_INSTALL' status='OK'/>. Stage completed. (Stage: Running first installation script on Host) >2012-11-08 07:36:36,062 INFO [org.ovirt.engine.core.bll.OVirtInstaller] (pool-4-thread-48) [2cb216e8] Installation of 10.70.36.2. Executing oVirt installation stage. (Stage: Downloading certificate request from Host) >2012-11-08 07:36:36,062 INFO [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-4-thread-48) SSH receive 10.70.36.2:'/tmp/cert_45ec7327-0abd-4564-9c77-f8c036cb475a.req' '/etc/pki/ovirt-engine/requests/cert_45ec7327-0abd-4564-9c77-f8c036cb475a.req') >2012-11-08 07:36:36,121 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-48) [2cb216e8] Installation of 10.70.36.2. successfully done sftp operation ( Stage: Downloading certificate request from Host) >2012-11-08 07:36:36,121 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-48) [2cb216e8] DownloadCertificateRequest ended:true >2012-11-08 07:36:36,121 INFO [org.ovirt.engine.core.bll.OVirtInstaller] (pool-4-thread-48) [2cb216e8] Installation of 10.70.36.2. Executing oVirt installation stage. (Stage: Sign certificate request and generate certificate) >2012-11-08 07:36:37,156 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-48) [2cb216e8] SignCertificateRequest ended:true >2012-11-08 07:36:37,156 INFO [org.ovirt.engine.core.bll.OVirtInstaller] (pool-4-thread-48) [2cb216e8] Installation of 10.70.36.2. Executing oVirt installation stage. (Stage: Upload signed sertificate to Host) >2012-11-08 07:36:37,173 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-48) [2cb216e8] Installation of 10.70.36.2. successfully done sftp operation ( Stage: Upload signed sertificate to Host) >2012-11-08 07:36:37,173 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-48) [2cb216e8] UploadSignedCertificate ended:true >2012-11-08 07:36:37,173 INFO [org.ovirt.engine.core.bll.OVirtInstaller] (pool-4-thread-48) [2cb216e8] Installation of 10.70.36.2. Executing oVirt installation stage. (Stage: Upload Cerficate Autority to Host) >2012-11-08 07:36:37,227 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-48) [2cb216e8] Installation of 10.70.36.2. successfully done sftp operation ( Stage: Upload Cerficate Autority to Host) >2012-11-08 07:36:37,228 INFO [org.ovirt.engine.core.bll.OVirtInstaller] (pool-4-thread-48) [2cb216e8] Installation of 10.70.36.2. Executing oVirt installation stage, sending SSH Command /usr/share/vdsm-reg/vdsm-complete -c 'ssl=true' 45ec7327-0abd-4564-9c77-f8c036cb475a 0. (Stage: Running second installation script on Host) >2012-11-08 07:36:37,228 INFO [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-4-thread-48) SSH execute 10.70.36.2 '/usr/share/vdsm-reg/vdsm-complete -c 'ssl=true' 45ec7327-0abd-4564-9c77-f8c036cb475a 0' >2012-11-08 07:36:38,364 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-99) Failed to decrypt Data must not be longer than 256 bytes >2012-11-08 07:36:38,365 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-5) Failed to decrypt Data must not be longer than 256 bytes >2012-11-08 07:36:39,997 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-43) Installation of 10.70.36.2. Received message: <BSTRAP component='instCert' status='OK'/>. FYI. (Stage: Running second installation script on Host) >2012-11-08 07:36:40,001 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-43) Installation of 10.70.36.2. Received message: <BSTRAP component='CoreDump' status='OK'/>. FYI. (Stage: Running second installation script on Host) >2012-11-08 07:36:40,017 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-43) Installation of 10.70.36.2. Received message: <BSTRAP component='cleanAll' status='OK'/>. FYI. (Stage: Running second installation script on Host) >2012-11-08 07:36:40,022 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-43) Installation of 10.70.36.2. Received message: <BSTRAP component='VDS Configuration' status='OK'/>. FYI. (Stage: Running second installation script on Host) >2012-11-08 07:36:40,027 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-43) Installation of 10.70.36.2. Received message: <BSTRAP component='Restart' status='OK' message='Restarting vdsmd service' />. FYI. (Stage: Running second installation script on Host) >2012-11-08 07:36:40,033 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-43) Installation of 10.70.36.2. Received message: <BSTRAP component='RHEV_INSTALL' status='OK'/>. Stage completed. (Stage: Running second installation script on Host) >2012-11-08 07:36:40,038 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-4-thread-48) [2cb216e8] After Installation pool-4-thread-48 >2012-11-08 07:36:40,047 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-48) [2cb216e8] START, SetVdsStatusVDSCommand(HostName = rhs-arch-srv2.lab.eng.blr.redhat.com, HostId = e9163546-299f-11e2-b43c-52540035ed48, status=NonResponsive, nonOperationalReason=NONE), log id: 6162f481 >2012-11-08 07:36:40,051 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-48) [2cb216e8] FINISH, SetVdsStatusVDSCommand, log id: 6162f481 >2012-11-08 07:36:40,053 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-48) [2cb216e8] START, SetVdsStatusVDSCommand(HostName = rhs-arch-srv2.lab.eng.blr.redhat.com, HostId = e9163546-299f-11e2-b43c-52540035ed48, status=Unassigned, nonOperationalReason=NONE), log id: 739915bf >2012-11-08 07:36:40,057 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-48) [2cb216e8] FINISH, SetVdsStatusVDSCommand, log id: 739915bf >2012-11-08 07:36:40,596 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-12) ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = e9163546-299f-11e2-b43c-52540035ed48 : rhs-arch-srv2.lab.eng.blr.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-11-08 07:36:43,302 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-18) [7228ffde] Running command: InitVdsOnUpCommand internal: true. >2012-11-08 07:36:43,308 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-18) [7228ffde] No string for UNASSIGNED type. Use default Log >2012-11-08 07:36:43,309 INFO [org.ovirt.engine.core.bll.MultipleActionsRunner] (pool-4-thread-48) [7228ffde] MultipleActionsRunner of type MigrateVmToServer invoked with no actions >2012-11-08 07:36:43,311 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-18) [2ef42033] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: e9163546-299f-11e2-b43c-52540035ed48 Type: VDS >2012-11-08 07:36:43,348 INFO [org.ovirt.engine.core.bll.UpdateVdsGroupCommand] (QuartzScheduler_Worker-18) [717d9ba3] Running command: UpdateVdsGroupCommand internal: true. Entities affected : ID: 99408929-82cf-4dc7-a532-9d998063fa95 Type: VdsGroups >2012-11-08 07:36:43,362 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-18) [7ab5f38b] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: e9163546-299f-11e2-b43c-52540035ed48 Type: VDS >2012-11-08 07:36:52,143 INFO [org.ovirt.engine.core.bll.MaintananceNumberOfVdssCommand] (pool-4-thread-48) [387332d3] Running command: MaintananceNumberOfVdssCommand internal: false. Entities affected : ID: e9163546-299f-11e2-b43c-52540035ed48 Type: VDS >2012-11-08 07:36:52,144 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-48) [387332d3] START, SetVdsStatusVDSCommand(HostName = rhs-arch-srv2.lab.eng.blr.redhat.com, HostId = e9163546-299f-11e2-b43c-52540035ed48, status=PreparingForMaintenance, nonOperationalReason=NONE), log id: 2b6895a4 >2012-11-08 07:36:52,148 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-48) [387332d3] FINISH, SetVdsStatusVDSCommand, log id: 2b6895a4 >2012-11-08 07:36:52,160 INFO [org.ovirt.engine.core.bll.MaintananceVdsCommand] (pool-4-thread-48) [387332d3] Running command: MaintananceVdsCommand internal: true. Entities affected : ID: e9163546-299f-11e2-b43c-52540035ed48 Type: VDS >2012-11-08 07:36:53,571 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-39) vds::Updated vds status from Preparing for Maintenance to Maintenance in database, vds = e9163546-299f-11e2-b43c-52540035ed48 : rhs-arch-srv2.lab.eng.blr.redhat.com >2012-11-08 07:36:53,579 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) Clearing cache of pool: 928c7030-2974-11e2-9380-52540035ed48 for problematic entities of VDS: rhs-arch-srv2.lab.eng.blr.redhat.com. >2012-11-08 07:40:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-85) Autorecovering hosts is disabled, skipping >2012-11-08 07:40:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-85) Autorecovering storage domains is disabled, skipping >2012-11-08 07:42:19,665 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-16) [525921d1] START, IsValidVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 52fc5ff0 >2012-11-08 07:42:19,665 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-16) [525921d1] FINISH, IsValidVDSCommand, return: true, log id: 52fc5ff0 >2012-11-08 07:42:19,676 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp-/127.0.0.1:8702-16) [525921d1] START, IsVmDuringInitiatingVDSCommand( vmId = 67bd2d56-8e0b-4328-bcfd-ea7f4758c885), log id: 1fe8ae17 >2012-11-08 07:42:19,676 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp-/127.0.0.1:8702-16) [525921d1] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 1fe8ae17 >2012-11-08 07:42:19,717 INFO [org.ovirt.engine.core.bll.RunVmOnceCommand] (ajp-/127.0.0.1:8702-16) [525921d1] Running command: RunVmOnceCommand internal: false. Entities affected : ID: 67bd2d56-8e0b-4328-bcfd-ea7f4758c885 Type: VM, ID: 67bd2d56-8e0b-4328-bcfd-ea7f4758c885 Type: VM >2012-11-08 07:42:19,775 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsoPrefixVDSCommand] (ajp-/127.0.0.1:8702-16) [525921d1] START, IsoPrefixVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3a4335e4 >2012-11-08 07:42:19,775 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsoPrefixVDSCommand] (ajp-/127.0.0.1:8702-16) [525921d1] FINISH, IsoPrefixVDSCommand, return: /rhev/data-center/3a17dcac-a6e3-4563-a4b0-f27d89b68dd6/7a268ac7-857a-4c53-966d-247deb509161/images/11111111-1111-1111-1111-111111111111, log id: 3a4335e4 >2012-11-08 07:42:19,789 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-16) [525921d1] START, CreateVmVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=67bd2d56-8e0b-4328-bcfd-ea7f4758c885, vm=org.ovirt.engine.core.common.businessentities.VM@2015e538), log id: 127a6467 >2012-11-08 07:42:19,808 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8702-16) [525921d1] START, CreateVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=67bd2d56-8e0b-4328-bcfd-ea7f4758c885, vm=org.ovirt.engine.core.common.businessentities.VM@2015e538), log id: 7c084a13 >2012-11-08 07:42:19,863 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8702-16) [525921d1] org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand spiceSslCipherSuite=DEFAULT,memSize=512,kvmEnable=true,smp=1,emulatedMachine=rhel6.3.0,vmType=kvm,keyboardLayout=en-us,nice=0,display=qxl,smpCoresPerSocket=1,spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard,timeOffset=-43200,transparentHugePages=true,vmId=67bd2d56-8e0b-4328-bcfd-ea7f4758c885,devices=[Ljava.util.Map;@79e0159e,acpiEnable=true,vmName=b1,cpuType=Nehalem,custom={viodiskcache=none} >2012-11-08 07:42:19,863 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8702-16) [525921d1] FINISH, CreateVDSCommand, log id: 7c084a13 >2012-11-08 07:42:19,864 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-16) [525921d1] IncreasePendingVms::CreateVmIncreasing vds rhs-gp-srv11.lab.eng.blr.redhat.com pending vcpu count, now 1. Vm: b1 >2012-11-08 07:42:19,868 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-16) [525921d1] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 127a6467 >2012-11-08 07:42:22,087 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (QuartzScheduler_Worker-7) START, DestroyVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=67bd2d56-8e0b-4328-bcfd-ea7f4758c885, force=false, secondsToWait=0, gracefully=false), log id: 34425706 >2012-11-08 07:42:22,142 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (QuartzScheduler_Worker-7) FINISH, DestroyVDSCommand, log id: 34425706 >2012-11-08 07:42:22,157 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-7) Running on vds during rerun failed vm: null >2012-11-08 07:42:22,158 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-7) vm b1 running in db and not running in vds - add to rerun treatment. vds rhs-gp-srv11.lab.eng.blr.redhat.com >2012-11-08 07:42:22,160 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-7) Rerun vm 67bd2d56-8e0b-4328-bcfd-ea7f4758c885. Called from vds rhs-gp-srv11.lab.eng.blr.redhat.com >2012-11-08 07:42:22,183 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsDynamicDataVDSCommand] (pool-4-thread-48) START, UpdateVdsDynamicDataVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vdsDynamic=org.ovirt.engine.core.common.businessentities.VdsDynamic@dbea1aad), log id: 7e715671 >2012-11-08 07:42:22,184 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsDynamicDataVDSCommand] (pool-4-thread-48) FINISH, UpdateVdsDynamicDataVDSCommand, log id: 7e715671 >2012-11-08 07:42:22,207 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (pool-4-thread-48) START, IsValidVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 77e29e1d >2012-11-08 07:42:22,208 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (pool-4-thread-48) FINISH, IsValidVDSCommand, return: true, log id: 77e29e1d >2012-11-08 07:42:22,224 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (pool-4-thread-48) START, IsVmDuringInitiatingVDSCommand( vmId = 67bd2d56-8e0b-4328-bcfd-ea7f4758c885), log id: 553458d2 >2012-11-08 07:42:22,224 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (pool-4-thread-48) FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 553458d2 >2012-11-08 07:42:22,258 INFO [org.ovirt.engine.core.bll.VdsSelector] (pool-4-thread-48) VDS rhs-gp-srv11.lab.eng.blr.redhat.com 96739be4-2990-11e2-8577-52540035ed48 is the same host the VM is currently running on VDS rhs-client11.lab.eng.blr.redhat.com 90b581c2-2977-11e2-82cd-52540035ed48 is not in up status or belongs to the VM's cluster VDS rhs-gp-srv4.lab.eng.blr.redhat.com d2bd32d4-2992-11e2-ab59-52540035ed48 is not in up status or belongs to the VM's cluster VDS rhs-arch-srv2.lab.eng.blr.redhat.com e9163546-299f-11e2-b43c-52540035ed48 is not in up status or belongs to the VM's cluster >2012-11-08 07:42:22,258 WARN [org.ovirt.engine.core.bll.RunVmOnceCommand] (pool-4-thread-48) CanDoAction of action RunVmOnce failed. Reasons:VAR__ACTION__RUN,VAR__TYPE__VM,$VmName b1,VAR__ACTION__RUN,VAR__TYPE__VM,ACTION_TYPE_FAILED_VDS_VM_CLUSTER >2012-11-08 07:43:33,328 INFO [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (ajp-/127.0.0.1:8702-5) SSH execute 10.70.36.2 'IDFILE=/etc/vdsm/vdsm.id; if [ -r "${IDFILE}" ]; then cat "${IDFILE}"; else UUID="$(dmidecode -s system-uuid 2> /dev/null | sed -e 's/.*Not.*//' )"; if [ -z "${UUID}" ]; then UUID="$(uuidgen 2> /dev/null)" && mkdir -p "$(dirname "${IDFILE}")" && echo "${UUID}" > "${IDFILE}" && chmod 0644 "${IDFILE}"; fi; [ -n "${UUID}" ] && echo "${UUID}"; fi' >2012-11-08 07:43:33,475 WARN [org.ovirt.engine.core.bll.AddVdsCommand] (ajp-/127.0.0.1:8702-5) [4941a8b4] CanDoAction of action AddVds failed. Reasons:VDS_REGISTER_UNIQUE_ID_AMBIGUOUS,$HostNameList rhs-arch-srv2.lab.eng.blr.redhat.com,VAR__ACTION__ADD,VAR__TYPE__HOST >2012-11-08 07:44:30,472 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsoPrefixVDSCommand] (ajp-/127.0.0.1:8702-11) [40dfba1f] START, IsoPrefixVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 400d5170 >2012-11-08 07:44:30,472 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsoPrefixVDSCommand] (ajp-/127.0.0.1:8702-11) [40dfba1f] FINISH, IsoPrefixVDSCommand, return: /rhev/data-center/3a17dcac-a6e3-4563-a4b0-f27d89b68dd6/7a268ac7-857a-4c53-966d-247deb509161/images/11111111-1111-1111-1111-111111111111, log id: 400d5170 >2012-11-08 07:44:30,533 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-11) [40dfba1f] START, IsValidVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: f58cc3c >2012-11-08 07:44:30,534 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-11) [40dfba1f] FINISH, IsValidVDSCommand, return: true, log id: f58cc3c >2012-11-08 07:44:30,552 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp-/127.0.0.1:8702-11) [40dfba1f] START, IsVmDuringInitiatingVDSCommand( vmId = 67bd2d56-8e0b-4328-bcfd-ea7f4758c885), log id: 332084ce >2012-11-08 07:44:30,552 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp-/127.0.0.1:8702-11) [40dfba1f] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 332084ce >2012-11-08 07:44:30,608 INFO [org.ovirt.engine.core.bll.RunVmOnceCommand] (ajp-/127.0.0.1:8702-11) [40dfba1f] Running command: RunVmOnceCommand internal: false. Entities affected : ID: 67bd2d56-8e0b-4328-bcfd-ea7f4758c885 Type: VM, ID: 67bd2d56-8e0b-4328-bcfd-ea7f4758c885 Type: VM >2012-11-08 07:44:30,687 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-11) [40dfba1f] START, CreateVmVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=67bd2d56-8e0b-4328-bcfd-ea7f4758c885, vm=org.ovirt.engine.core.common.businessentities.VM@403ec5ef), log id: 517a4342 >2012-11-08 07:44:30,710 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8702-11) [40dfba1f] START, CreateVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=67bd2d56-8e0b-4328-bcfd-ea7f4758c885, vm=org.ovirt.engine.core.common.businessentities.VM@403ec5ef), log id: 7021f37d >2012-11-08 07:44:30,764 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8702-11) [40dfba1f] org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand spiceSslCipherSuite=DEFAULT,memSize=512,kvmEnable=true,smp=1,emulatedMachine=rhel6.3.0,vmType=kvm,keyboardLayout=en-us,nice=0,display=qxl,smpCoresPerSocket=1,spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard,timeOffset=-43200,transparentHugePages=true,vmId=67bd2d56-8e0b-4328-bcfd-ea7f4758c885,devices=[Ljava.util.Map;@3741d283,acpiEnable=true,vmName=b1,cpuType=Nehalem,custom={viodiskcache=none} >2012-11-08 07:44:30,764 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8702-11) [40dfba1f] FINISH, CreateVDSCommand, log id: 7021f37d >2012-11-08 07:44:30,765 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-11) [40dfba1f] IncreasePendingVms::CreateVmIncreasing vds rhs-gp-srv11.lab.eng.blr.redhat.com pending vcpu count, now 1. Vm: b1 >2012-11-08 07:44:30,769 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-11) [40dfba1f] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 517a4342 >2012-11-08 07:44:33,099 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (QuartzScheduler_Worker-16) START, DestroyVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=67bd2d56-8e0b-4328-bcfd-ea7f4758c885, force=false, secondsToWait=0, gracefully=false), log id: 11599ec8 >2012-11-08 07:44:33,156 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (QuartzScheduler_Worker-16) FINISH, DestroyVDSCommand, log id: 11599ec8 >2012-11-08 07:44:33,173 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-16) Running on vds during rerun failed vm: null >2012-11-08 07:44:33,174 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-16) vm b1 running in db and not running in vds - add to rerun treatment. vds rhs-gp-srv11.lab.eng.blr.redhat.com >2012-11-08 07:44:33,179 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-16) Rerun vm 67bd2d56-8e0b-4328-bcfd-ea7f4758c885. Called from vds rhs-gp-srv11.lab.eng.blr.redhat.com >2012-11-08 07:44:33,207 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsDynamicDataVDSCommand] (pool-4-thread-47) START, UpdateVdsDynamicDataVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vdsDynamic=org.ovirt.engine.core.common.businessentities.VdsDynamic@dbea1aad), log id: e49522f >2012-11-08 07:44:33,209 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsDynamicDataVDSCommand] (pool-4-thread-47) FINISH, UpdateVdsDynamicDataVDSCommand, log id: e49522f >2012-11-08 07:44:33,231 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (pool-4-thread-47) START, IsValidVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6c9956a5 >2012-11-08 07:44:33,232 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (pool-4-thread-47) FINISH, IsValidVDSCommand, return: true, log id: 6c9956a5 >2012-11-08 07:44:33,249 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (pool-4-thread-47) START, IsVmDuringInitiatingVDSCommand( vmId = 67bd2d56-8e0b-4328-bcfd-ea7f4758c885), log id: 7d7c5fb9 >2012-11-08 07:44:33,249 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (pool-4-thread-47) FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 7d7c5fb9 >2012-11-08 07:44:33,287 INFO [org.ovirt.engine.core.bll.VdsSelector] (pool-4-thread-47) VDS rhs-gp-srv11.lab.eng.blr.redhat.com 96739be4-2990-11e2-8577-52540035ed48 is the same host the VM is currently running on VDS rhs-client11.lab.eng.blr.redhat.com 90b581c2-2977-11e2-82cd-52540035ed48 is not in up status or belongs to the VM's cluster VDS rhs-gp-srv4.lab.eng.blr.redhat.com d2bd32d4-2992-11e2-ab59-52540035ed48 is not in up status or belongs to the VM's cluster VDS rhs-arch-srv2.lab.eng.blr.redhat.com e9163546-299f-11e2-b43c-52540035ed48 is not in up status or belongs to the VM's cluster >2012-11-08 07:44:33,287 WARN [org.ovirt.engine.core.bll.RunVmOnceCommand] (pool-4-thread-47) CanDoAction of action RunVmOnce failed. Reasons:VAR__ACTION__RUN,VAR__TYPE__VM,$VmName b1,VAR__ACTION__RUN,VAR__TYPE__VM,ACTION_TYPE_FAILED_VDS_VM_CLUSTER >2012-11-08 07:45:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-80) Autorecovering hosts is disabled, skipping >2012-11-08 07:45:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-80) Autorecovering storage domains is disabled, skipping >2012-11-08 07:46:36,682 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-6) Failed to decrypt Data must not be longer than 256 bytes >2012-11-08 07:46:36,695 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-17) Failed to decrypt Data must start with zero >2012-11-08 07:47:51,187 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-80) Failed to decrypt Data must not be longer than 256 bytes >2012-11-08 07:47:51,187 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-67) Failed to decrypt Data must not be longer than 256 bytes >2012-11-08 07:47:57,219 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-96) Failed to decrypt Data must not be longer than 256 bytes >2012-11-08 07:47:57,219 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-92) Failed to decrypt Data must not be longer than 256 bytes >2012-11-08 07:48:33,455 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-80) Failed to decrypt Data must not be longer than 256 bytes >2012-11-08 07:48:33,455 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-75) Failed to decrypt Data must not be longer than 256 bytes >2012-11-08 07:49:21,775 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-91) Failed to decrypt Data must not be longer than 256 bytes >2012-11-08 07:49:21,775 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-94) Failed to decrypt Data must not be longer than 256 bytes >2012-11-08 07:49:22,646 INFO [org.ovirt.engine.core.bll.RemoveVdsCommand] (ajp-/127.0.0.1:8702-5) Lock Acquired to object EngineLock [exclusiveLocks= key: e9163546-299f-11e2-b43c-52540035ed48 value: VDS >, sharedLocks= ] >2012-11-08 07:49:22,656 INFO [org.ovirt.engine.core.bll.RemoveVdsCommand] (pool-4-thread-48) [3a1d8c33] Running command: RemoveVdsCommand internal: false. Entities affected : ID: e9163546-299f-11e2-b43c-52540035ed48 Type: VDS >2012-11-08 07:49:22,695 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (pool-4-thread-48) [3a1d8c33] START, RemoveVdsVDSCommand( HostId = e9163546-299f-11e2-b43c-52540035ed48), log id: 22693399 >2012-11-08 07:49:22,695 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (pool-4-thread-48) [3a1d8c33] vdsManager::disposing >2012-11-08 07:49:22,695 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (pool-4-thread-48) [3a1d8c33] FINISH, RemoveVdsVDSCommand, log id: 22693399 >2012-11-08 07:49:22,696 INFO [org.ovirt.engine.core.bll.RemoveVdsCommand] (pool-4-thread-48) [3a1d8c33] Lock freed to object EngineLock [exclusiveLocks= key: e9163546-299f-11e2-b43c-52540035ed48 value: VDS >, sharedLocks= ] >2012-11-08 07:49:57,849 INFO [org.ovirt.engine.core.bll.LogoutUserCommand] (ajp-/127.0.0.1:8702-14) [5cbe06f7] Running command: LogoutUserCommand internal: false. >2012-11-08 07:50:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-76) Autorecovering hosts is disabled, skipping >2012-11-08 07:50:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-76) Autorecovering storage domains is disabled, skipping >2012-11-08 07:50:00,734 INFO [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (ajp-/127.0.0.1:8702-2) SSH execute 10.70.36.2 'IDFILE=/etc/vdsm/vdsm.id; if [ -r "${IDFILE}" ]; then cat "${IDFILE}"; else UUID="$(dmidecode -s system-uuid 2> /dev/null | sed -e 's/.*Not.*//' )"; if [ -z "${UUID}" ]; then UUID="$(uuidgen 2> /dev/null)" && mkdir -p "$(dirname "${IDFILE}")" && echo "${UUID}" > "${IDFILE}" && chmod 0644 "${IDFILE}"; fi; [ -n "${UUID}" ] && echo "${UUID}"; fi' >2012-11-08 07:50:00,882 INFO [org.ovirt.engine.core.bll.AddVdsCommand] (ajp-/127.0.0.1:8702-2) [641bce7b] Running command: AddVdsCommand internal: false. Entities affected : ID: f9f22222-2976-11e2-a2cd-52540035ed48 Type: VdsGroups >2012-11-08 07:50:00,905 INFO [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp-/127.0.0.1:8702-2) [6f0561d8] Running command: AddVdsSpmIdCommand internal: true. Entities affected : ID: cf017816-29a2-11e2-949e-52540035ed48 Type: VDS >2012-11-08 07:50:00,908 INFO [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp-/127.0.0.1:8702-2) [6f0561d8] Lock freed to object EngineLock [exclusiveLocks= key: f4c98a0b-0578-485f-90f4-743df3d05d7e value: REGISTER_VDS >, sharedLocks= ] >2012-11-08 07:50:00,909 ERROR [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp-/127.0.0.1:8702-2) [6f0561d8] Cannot get vdsManager for vdsid=cf017816-29a2-11e2-949e-52540035ed48 >2012-11-08 07:50:00,909 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (ajp-/127.0.0.1:8702-2) [6f0561d8] START, RemoveVdsVDSCommand(HostName = rhs-arch-srv2.lab.eng.blr.redhat.com, HostId = cf017816-29a2-11e2-949e-52540035ed48), log id: 3e67290 >2012-11-08 07:50:00,910 ERROR [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp-/127.0.0.1:8702-2) [6f0561d8] Cannot get vdsManager for vdsid=cf017816-29a2-11e2-949e-52540035ed48 >2012-11-08 07:50:00,910 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (ajp-/127.0.0.1:8702-2) [6f0561d8] FINISH, RemoveVdsVDSCommand, log id: 3e67290 >2012-11-08 07:50:00,910 ERROR [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp-/127.0.0.1:8702-2) [6f0561d8] Cannot get vdsManager for vdsid=cf017816-29a2-11e2-949e-52540035ed48 >2012-11-08 07:50:00,911 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-2) [6f0561d8] START, AddVdsVDSCommand(HostName = rhs-arch-srv2.lab.eng.blr.redhat.com, HostId = cf017816-29a2-11e2-949e-52540035ed48), log id: 668f0dcb >2012-11-08 07:50:00,911 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-2) [6f0561d8] AddVds - entered , starting logic to add VDS cf017816-29a2-11e2-949e-52540035ed48 >2012-11-08 07:50:00,912 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-2) [6f0561d8] AddVds - VDS cf017816-29a2-11e2-949e-52540035ed48 was added, will try to add it to the resource manager >2012-11-08 07:50:00,913 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ajp-/127.0.0.1:8702-2) [6f0561d8] Eneterd VdsManager:constructor >2012-11-08 07:50:00,913 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ajp-/127.0.0.1:8702-2) [6f0561d8] vdsBroker(10.70.36.2,54,321) >2012-11-08 07:50:00,913 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp-/127.0.0.1:8702-2) [6f0561d8] ResourceManager::AddVds - VDS cf017816-29a2-11e2-949e-52540035ed48 was added to the Resource Manager >2012-11-08 07:50:00,913 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-2) [6f0561d8] FINISH, AddVdsVDSCommand, log id: 668f0dcb >2012-11-08 07:50:00,930 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-4-thread-48) [4860102e] Running command: InstallVdsCommand internal: true. Entities affected : ID: cf017816-29a2-11e2-949e-52540035ed48 Type: VDS >2012-11-08 07:50:00,932 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-4-thread-48) [4860102e] Before Installation pool-4-thread-48 >2012-11-08 07:50:00,932 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-48) [4860102e] Installation of 10.70.36.2. Executing installation stage. (Stage: Starting Host installation) >2012-11-08 07:50:00,932 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-48) [4860102e] Installation of 10.70.36.2. Executing installation stage. (Stage: Connecting to Host) >2012-11-08 07:50:00,996 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-48) [4860102e] Installation of 10.70.36.2. Received message: <BSTRAP component='RHEV_INSTALL' status='OK' message='Connected to Host 10.70.36.2 with SSH key fingerprint: d3:61:44:22:06:7a:76:2a:e3:12:7e:8e:41:56:25:a0'/>. FYI. (Stage: Connecting to Host) >2012-11-08 07:50:01,025 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-48) [4860102e] Installation of 10.70.36.2. Successfully connected to server ssh. (Stage: Connecting to Host) >2012-11-08 07:50:01,025 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-48) [4860102e] Installation of 10.70.36.2. Executing installation stage. (Stage: Get the unique vds id) >2012-11-08 07:50:01,025 INFO [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-4-thread-48) SSH execute 10.70.36.2 'IDFILE=/etc/vdsm/vdsm.id; if [ -r "${IDFILE}" ]; then cat "${IDFILE}"; else UUID="$(dmidecode -s system-uuid 2> /dev/null | sed -e 's/.*Not.*//' )"; if [ -z "${UUID}" ]; then UUID="$(uuidgen 2> /dev/null)" && mkdir -p "$(dirname "${IDFILE}")" && echo "${UUID}" > "${IDFILE}" && chmod 0644 "${IDFILE}"; fi; [ -n "${UUID}" ] && echo "${UUID}"; fi' >2012-11-08 07:50:01,144 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-52) Installation of 10.70.36.2. Received message: 00000000-0000-0000-0000-0025902F9398. FYI. (Stage: Get the unique vds id) >2012-11-08 07:50:01,146 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-52) Installation of 10.70.36.2. Assigning unique id 00000000-0000-0000-0000-0025902F9398 to Host. (Stage: Get the unique vds id) >2012-11-08 07:50:01,150 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-48) [4860102e] Installation of 10.70.36.2. Executing installation stage. (Stage: Upload Installation script to Host) >2012-11-08 07:50:01,150 INFO [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-4-thread-48) SSH execute 10.70.36.2 'cat > '/tmp/ovirt-id_rsa_cc228197-deab-4ad9-bbfe-867cdc58f559'' >2012-11-08 07:50:01,234 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-48) [4860102e] Installation of 10.70.36.2. Executing installation stage. (Stage: Running first installation script on Host) >2012-11-08 07:50:01,235 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-48) [4860102e] Installation of 10.70.36.2. Sending SSH Command umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar -C "${MYTMP}" --no-same-permissions -o -x && "${MYTMP}"/setup -c 'ssl=true;management_port=54321' -O 'lab.eng.blr.redhat.com' -t 2012-11-08T12:50:01 -S /tmp/ovirt-id_rsa_cc228197-deab-4ad9-bbfe-867cdc58f559 -p 80 -b -B rhevm http://guido.lab.eng.blr.redhat.com:80/Components/vds/ http://guido.lab.eng.blr.redhat.com:80/Components/vds/ 10.70.36.2 cc228197-deab-4ad9-bbfe-867cdc58f559 False < /var/cache/ovirt-engine/vdsm-bootstrap-2.tar. (Stage: Running first installation script on Host) >2012-11-08 07:50:01,240 INFO [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-4-thread-48) SSH execute 10.70.36.2 'umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar -C "${MYTMP}" --no-same-permissions -o -x && "${MYTMP}"/setup -c 'ssl=true;management_port=54321' -O 'lab.eng.blr.redhat.com' -t 2012-11-08T12:50:01 -S /tmp/ovirt-id_rsa_cc228197-deab-4ad9-bbfe-867cdc58f559 -p 80 -b -B rhevm http://guido.lab.eng.blr.redhat.com:80/Components/vds/ http://guido.lab.eng.blr.redhat.com:80/Components/vds/ 10.70.36.2 cc228197-deab-4ad9-bbfe-867cdc58f559 False' >2012-11-08 07:50:01,340 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-52) Installation of 10.70.36.2. Received message: <BSTRAP component='RHEV_INSTALL' status='OK' message='oVirt Node DETECTED'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 07:50:02,041 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-70) Failed to decrypt Data must not be longer than 256 bytes >2012-11-08 07:50:02,041 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-74) Failed to decrypt Data must not be longer than 256 bytes >2012-11-08 07:50:02,895 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-52) Installation of 10.70.36.2. Received message: <BSTRAP component='RHEV_INSTALL' status='OK' message='RHEV-H ACCESSIBLE'/>. Stage completed. (Stage: Running first installation script on Host) >2012-11-08 07:50:02,935 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-48) [4860102e] Script ended, result is {1} >2012-11-08 07:50:02,935 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-4-thread-48) [4860102e] After Installation pool-4-thread-48 >2012-11-08 07:50:03,070 WARN [org.ovirt.engine.core.dal.dbbroker.generic.DBConfigUtils] (ajp-/127.0.0.1:8702-4) Adding new value to configuration cache. >2012-11-08 07:50:03,070 WARN [org.ovirt.engine.core.dal.dbbroker.generic.DBConfigUtils] (ajp-/127.0.0.1:8702-4) Didnt find the value of OtpExpirationInSeconds in DB for version general - using default: 600 >2012-11-08 07:50:03,071 INFO [org.ovirt.engine.core.bll.RegisterVdsQuery] (ajp-/127.0.0.1:8702-4) Running Command: RegisterVds >2012-11-08 07:50:03,074 INFO [org.ovirt.engine.core.bll.ApproveVdsCommand] (pool-4-thread-48) [19131f95] Running command: ApproveVdsCommand internal: true. Entities affected : ID: cf017816-29a2-11e2-949e-52540035ed48 Type: VDS >2012-11-08 07:50:03,074 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-4-thread-48) [19131f95] Before Installation pool-4-thread-48, Powerclient/oVirtNode case: setting status to installing >2012-11-08 07:50:03,075 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-48) [19131f95] START, SetVdsStatusVDSCommand(HostName = rhs-arch-srv2.lab.eng.blr.redhat.com, HostId = cf017816-29a2-11e2-949e-52540035ed48, status=Installing, nonOperationalReason=NONE), log id: 6c644a39 >2012-11-08 07:50:03,080 INFO [org.ovirt.engine.core.register.RegisterServlet] (ajp-/127.0.0.1:8702-4) Succeeded to run RegisterVds. >2012-11-08 07:50:03,080 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-48) [19131f95] FINISH, SetVdsStatusVDSCommand, log id: 6c644a39 >2012-11-08 07:50:03,081 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-4-thread-48) [19131f95] Before Installation pool-4-thread-48 >2012-11-08 07:50:03,081 INFO [org.ovirt.engine.core.bll.OVirtInstaller] (pool-4-thread-48) [19131f95] Installation of 10.70.36.2. Executing oVirt installation stage. (Stage: Starting Host installation) >2012-11-08 07:50:03,082 INFO [org.ovirt.engine.core.bll.OVirtInstaller] (pool-4-thread-48) [19131f95] Installation of 10.70.36.2. Executing oVirt installation stage. (Stage: Connecting to Host) >2012-11-08 07:50:03,150 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-48) [19131f95] Installation of 10.70.36.2. Received message: <BSTRAP component='RHEV_INSTALL' status='OK' message='Connected to Host 10.70.36.2 with SSH key fingerprint: d3:61:44:22:06:7a:76:2a:e3:12:7e:8e:41:56:25:a0'/>. FYI. (Stage: Connecting to Host) >2012-11-08 07:50:03,173 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-48) [19131f95] Installation of 10.70.36.2. Successfully connected to server ssh. (Stage: Connecting to Host) >2012-11-08 07:50:03,173 INFO [org.ovirt.engine.core.bll.OVirtInstaller] (pool-4-thread-48) [19131f95] Installation of 10.70.36.2. Executing oVirt installation stage. (Stage: Running first installation script on Host) >2012-11-08 07:50:03,173 INFO [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-4-thread-48) SSH execute 10.70.36.2 '/usr/share/vdsm-reg/vdsm-gen-cert -O "lab.eng.blr.redhat.com" 10.70.36.2 8c637340-5188-47cc-8747-af81f5c9c64c' >2012-11-08 07:50:04,917 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-80) Initializing Host: rhs-arch-srv2.lab.eng.blr.redhat.com >2012-11-08 07:50:05,933 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-56) Installation of 10.70.36.2. Received message: <BSTRAP component='Encryption setup' status='OK'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 07:50:05,940 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-56) Installation of 10.70.36.2. Received message: <BSTRAP component='RHEV_INSTALL' status='OK'/>. Stage completed. (Stage: Running first installation script on Host) >2012-11-08 07:50:05,963 INFO [org.ovirt.engine.core.bll.OVirtInstaller] (pool-4-thread-48) [19131f95] Installation of 10.70.36.2. Executing oVirt installation stage. (Stage: Downloading certificate request from Host) >2012-11-08 07:50:05,963 INFO [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-4-thread-48) SSH receive 10.70.36.2:'/tmp/cert_8c637340-5188-47cc-8747-af81f5c9c64c.req' '/etc/pki/ovirt-engine/requests/cert_8c637340-5188-47cc-8747-af81f5c9c64c.req') >2012-11-08 07:50:06,018 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-48) [19131f95] Installation of 10.70.36.2. successfully done sftp operation ( Stage: Downloading certificate request from Host) >2012-11-08 07:50:06,018 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-48) [19131f95] DownloadCertificateRequest ended:true >2012-11-08 07:50:06,018 INFO [org.ovirt.engine.core.bll.OVirtInstaller] (pool-4-thread-48) [19131f95] Installation of 10.70.36.2. Executing oVirt installation stage. (Stage: Sign certificate request and generate certificate) >2012-11-08 07:50:06,057 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-75) Failed to decrypt Data must not be longer than 256 bytes >2012-11-08 07:50:06,057 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-67) Failed to decrypt Data must not be longer than 256 bytes >2012-11-08 07:50:07,021 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-48) [19131f95] SignCertificateRequest ended:true >2012-11-08 07:50:07,021 INFO [org.ovirt.engine.core.bll.OVirtInstaller] (pool-4-thread-48) [19131f95] Installation of 10.70.36.2. Executing oVirt installation stage. (Stage: Upload signed sertificate to Host) >2012-11-08 07:50:07,036 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-48) [19131f95] Installation of 10.70.36.2. successfully done sftp operation ( Stage: Upload signed sertificate to Host) >2012-11-08 07:50:07,036 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-48) [19131f95] UploadSignedCertificate ended:true >2012-11-08 07:50:07,036 INFO [org.ovirt.engine.core.bll.OVirtInstaller] (pool-4-thread-48) [19131f95] Installation of 10.70.36.2. Executing oVirt installation stage. (Stage: Upload Cerficate Autority to Host) >2012-11-08 07:50:07,094 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-48) [19131f95] Installation of 10.70.36.2. successfully done sftp operation ( Stage: Upload Cerficate Autority to Host) >2012-11-08 07:50:07,094 INFO [org.ovirt.engine.core.bll.OVirtInstaller] (pool-4-thread-48) [19131f95] Installation of 10.70.36.2. Executing oVirt installation stage, sending SSH Command /usr/share/vdsm-reg/vdsm-complete -c 'ssl=true' 8c637340-5188-47cc-8747-af81f5c9c64c 0. (Stage: Running second installation script on Host) >2012-11-08 07:50:07,094 INFO [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-4-thread-48) SSH execute 10.70.36.2 '/usr/share/vdsm-reg/vdsm-complete -c 'ssl=true' 8c637340-5188-47cc-8747-af81f5c9c64c 0' >2012-11-08 07:50:08,060 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-81) Failed to decrypt Data must not be longer than 256 bytes >2012-11-08 07:50:08,060 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-90) Failed to decrypt Data must not be longer than 256 bytes >2012-11-08 07:50:10,693 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-56) Installation of 10.70.36.2. Received message: <BSTRAP component='instCert' status='OK'/>. FYI. (Stage: Running second installation script on Host) >2012-11-08 07:50:10,698 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-56) Installation of 10.70.36.2. Received message: <BSTRAP component='CoreDump' status='OK'/>. FYI. (Stage: Running second installation script on Host) >2012-11-08 07:50:10,702 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-56) Installation of 10.70.36.2. Received message: <BSTRAP component='cleanAll' status='OK'/>. FYI. (Stage: Running second installation script on Host) >2012-11-08 07:50:10,706 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-56) Installation of 10.70.36.2. Received message: <BSTRAP component='VDS Configuration' status='OK'/>. FYI. (Stage: Running second installation script on Host) >2012-11-08 07:50:10,710 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-56) Installation of 10.70.36.2. Received message: <BSTRAP component='Restart' status='OK' message='Restarting vdsmd service' />. FYI. (Stage: Running second installation script on Host) >2012-11-08 07:50:10,715 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-56) Installation of 10.70.36.2. Received message: <BSTRAP component='RHEV_INSTALL' status='OK'/>. Stage completed. (Stage: Running second installation script on Host) >2012-11-08 07:50:10,724 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-4-thread-48) [19131f95] After Installation pool-4-thread-48 >2012-11-08 07:50:10,726 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-48) [19131f95] START, SetVdsStatusVDSCommand(HostName = rhs-arch-srv2.lab.eng.blr.redhat.com, HostId = cf017816-29a2-11e2-949e-52540035ed48, status=NonResponsive, nonOperationalReason=NONE), log id: 3653b6e8 >2012-11-08 07:50:10,731 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-48) [19131f95] FINISH, SetVdsStatusVDSCommand, log id: 3653b6e8 >2012-11-08 07:50:10,733 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-48) [19131f95] START, SetVdsStatusVDSCommand(HostName = rhs-arch-srv2.lab.eng.blr.redhat.com, HostId = cf017816-29a2-11e2-949e-52540035ed48, status=Unassigned, nonOperationalReason=NONE), log id: 5f20c27c >2012-11-08 07:50:10,738 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-48) [19131f95] FINISH, SetVdsStatusVDSCommand, log id: 5f20c27c >2012-11-08 07:50:10,742 INFO [org.ovirt.engine.core.bll.RegisterVdsQuery] (pool-4-thread-48) [19131f95] Approval of oVirt cf017816-29a2-11e2-949e-52540035ed48 ended successefully. >2012-11-08 07:50:10,931 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-94) ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = cf017816-29a2-11e2-949e-52540035ed48 : rhs-arch-srv2.lab.eng.blr.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-11-08 07:50:13,645 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-93) [7e398edb] Running command: InitVdsOnUpCommand internal: true. >2012-11-08 07:50:13,654 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-93) [7e398edb] No string for UNASSIGNED type. Use default Log >2012-11-08 07:50:13,655 INFO [org.ovirt.engine.core.bll.MultipleActionsRunner] (pool-4-thread-48) [7e398edb] MultipleActionsRunner of type MigrateVmToServer invoked with no actions >2012-11-08 07:50:13,660 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-93) [2eae715d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: cf017816-29a2-11e2-949e-52540035ed48 Type: VDS >2012-11-08 07:50:13,670 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-93) [23ef92ac] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: cf017816-29a2-11e2-949e-52540035ed48 Type: VDS >2012-11-08 07:50:32,218 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-40) Failed to decrypt Data must not be longer than 256 bytes >2012-11-08 07:50:32,228 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-44) Failed to decrypt Data must start with zero >2012-11-08 07:50:38,256 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-54) Failed to decrypt Data must not be longer than 256 bytes >2012-11-08 07:50:38,267 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-46) Failed to decrypt Data must start with zero >2012-11-08 07:50:58,771 INFO [org.ovirt.engine.core.bll.LoginAdminUserCommand] (ajp-/127.0.0.1:8702-14) Checking if user admin@internal is an admin, result true >2012-11-08 07:50:58,771 INFO [org.ovirt.engine.core.bll.LoginAdminUserCommand] (ajp-/127.0.0.1:8702-14) Running command: LoginAdminUserCommand internal: false. >2012-11-08 07:51:08,456 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-25) Failed to decrypt Data must not be longer than 256 bytes >2012-11-08 07:51:08,456 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-27) Failed to decrypt Data must not be longer than 256 bytes >2012-11-08 07:52:18,956 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-84) Failed to decrypt Data must not be longer than 256 bytes >2012-11-08 07:52:18,965 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-86) Failed to decrypt Data must start with zero >2012-11-08 07:52:33,046 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-22) Failed to decrypt Data must not be longer than 256 bytes >2012-11-08 07:52:33,046 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-23) Failed to decrypt Data must not be longer than 256 bytes >2012-11-08 07:52:35,049 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-27) Failed to decrypt Data must not be longer than 256 bytes >2012-11-08 07:52:35,049 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-28) Failed to decrypt Data must not be longer than 256 bytes >2012-11-08 07:52:51,679 ERROR [org.ovirt.engine.core.bll.GetoVirtISOsQuery] (ajp-/127.0.0.1:8702-9) ovirt ISOs directory not found. Search in: /usr/share/rhev-hypervisor >2012-11-08 07:52:53,156 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-64) Failed to decrypt Data must not be longer than 256 bytes >2012-11-08 07:52:53,156 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-76) Failed to decrypt Data must not be longer than 256 bytes >2012-11-08 07:53:01,195 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-84) Failed to decrypt Data must not be longer than 256 bytes >2012-11-08 07:53:01,195 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-86) Failed to decrypt Data must not be longer than 256 bytes >2012-11-08 07:53:19,314 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-26) Failed to decrypt Data must not be longer than 256 bytes >2012-11-08 07:53:19,314 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-21) Failed to decrypt Data must not be longer than 256 bytes >2012-11-08 07:53:21,316 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-33) Failed to decrypt Data must not be longer than 256 bytes >2012-11-08 07:53:21,316 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-41) Failed to decrypt Data must not be longer than 256 bytes >2012-11-08 07:53:30,102 ERROR [org.ovirt.engine.core.bll.FencingExecutor] (ajp-/127.0.0.1:8702-13) Failed to run Power Management command on Host , no running proxy Host was found. >2012-11-08 07:53:38,336 ERROR [org.ovirt.engine.core.bll.FencingExecutor] (ajp-/127.0.0.1:8702-14) Failed to run Power Management command on Host , no running proxy Host was found. >2012-11-08 07:53:44,493 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsoPrefixVDSCommand] (ajp-/127.0.0.1:8702-2) [a1b085b] START, IsoPrefixVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 2cd3b3ca >2012-11-08 07:53:44,493 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsoPrefixVDSCommand] (ajp-/127.0.0.1:8702-2) [a1b085b] FINISH, IsoPrefixVDSCommand, return: /rhev/data-center/3a17dcac-a6e3-4563-a4b0-f27d89b68dd6/7a268ac7-857a-4c53-966d-247deb509161/images/11111111-1111-1111-1111-111111111111, log id: 2cd3b3ca >2012-11-08 07:53:44,517 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-2) [a1b085b] START, IsValidVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3365d611 >2012-11-08 07:53:44,518 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-2) [a1b085b] FINISH, IsValidVDSCommand, return: true, log id: 3365d611 >2012-11-08 07:53:44,535 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp-/127.0.0.1:8702-2) [a1b085b] START, IsVmDuringInitiatingVDSCommand( vmId = 68655de5-7bd9-4bdc-8628-3b2069ec312f), log id: 11b6e42f >2012-11-08 07:53:44,535 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp-/127.0.0.1:8702-2) [a1b085b] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 11b6e42f >2012-11-08 07:53:44,583 INFO [org.ovirt.engine.core.bll.RunVmOnceCommand] (ajp-/127.0.0.1:8702-2) [a1b085b] Running command: RunVmOnceCommand internal: false. Entities affected : ID: 68655de5-7bd9-4bdc-8628-3b2069ec312f Type: VM, ID: 68655de5-7bd9-4bdc-8628-3b2069ec312f Type: VM >2012-11-08 07:53:44,664 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-2) [a1b085b] START, CreateVmVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=68655de5-7bd9-4bdc-8628-3b2069ec312f, vm=org.ovirt.engine.core.common.businessentities.VM@65c7e4b3), log id: 6993a37 >2012-11-08 07:53:44,678 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8702-2) [a1b085b] START, CreateVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=68655de5-7bd9-4bdc-8628-3b2069ec312f, vm=org.ovirt.engine.core.common.businessentities.VM@65c7e4b3), log id: 2fe48f86 >2012-11-08 07:53:44,721 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8702-2) [a1b085b] org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand spiceSslCipherSuite=DEFAULT,memSize=512,kvmEnable=true,smp=1,emulatedMachine=rhel6.3.0,vmType=kvm,keyboardLayout=en-us,nice=0,display=qxl,smpCoresPerSocket=1,spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard,timeOffset=-43200,transparentHugePages=true,vmId=68655de5-7bd9-4bdc-8628-3b2069ec312f,devices=[Ljava.util.Map;@1579fd10,acpiEnable=true,vmName=b2,cpuType=Nehalem,custom={viodiskcache=writeback} >2012-11-08 07:53:44,721 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8702-2) [a1b085b] FINISH, CreateVDSCommand, log id: 2fe48f86 >2012-11-08 07:53:44,721 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-2) [a1b085b] IncreasePendingVms::CreateVmIncreasing vds rhs-gp-srv11.lab.eng.blr.redhat.com pending vcpu count, now 1. Vm: b2 >2012-11-08 07:53:44,723 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-2) [a1b085b] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 6993a37 >2012-11-08 07:53:46,693 ERROR [org.ovirt.engine.core.bll.FencingExecutor] (ajp-/127.0.0.1:8702-4) Failed to run Power Management command on Host , no running proxy Host was found. >2012-11-08 07:53:47,211 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (QuartzScheduler_Worker-62) START, DestroyVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=68655de5-7bd9-4bdc-8628-3b2069ec312f, force=false, secondsToWait=0, gracefully=false), log id: 52c5abfc >2012-11-08 07:53:47,268 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (QuartzScheduler_Worker-62) FINISH, DestroyVDSCommand, log id: 52c5abfc >2012-11-08 07:53:47,288 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-62) Running on vds during rerun failed vm: null >2012-11-08 07:53:47,289 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-62) vm b2 running in db and not running in vds - add to rerun treatment. vds rhs-gp-srv11.lab.eng.blr.redhat.com >2012-11-08 07:53:47,289 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (QuartzScheduler_Worker-62) START, FullListVdsCommand( HostId = 96739be4-2990-11e2-8577-52540035ed48, vds=null, vmIds=[68655de5-7bd9-4bdc-8628-3b2069ec312f]), log id: 6c3c3cc2 >2012-11-08 07:53:47,308 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (QuartzScheduler_Worker-62) FINISH, FullListVdsCommand, return: [Lorg.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcStruct;@66b91c11, log id: 6c3c3cc2 >2012-11-08 07:53:47,311 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-62) Rerun vm 68655de5-7bd9-4bdc-8628-3b2069ec312f. Called from vds rhs-gp-srv11.lab.eng.blr.redhat.com >2012-11-08 07:53:47,336 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsDynamicDataVDSCommand] (pool-4-thread-48) START, UpdateVdsDynamicDataVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vdsDynamic=org.ovirt.engine.core.common.businessentities.VdsDynamic@dbea1aad), log id: c056dd5 >2012-11-08 07:53:47,338 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsDynamicDataVDSCommand] (pool-4-thread-48) FINISH, UpdateVdsDynamicDataVDSCommand, log id: c056dd5 >2012-11-08 07:53:47,354 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (pool-4-thread-48) START, IsValidVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 7aa2c9be >2012-11-08 07:53:47,355 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (pool-4-thread-48) FINISH, IsValidVDSCommand, return: true, log id: 7aa2c9be >2012-11-08 07:53:47,369 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (pool-4-thread-48) START, IsVmDuringInitiatingVDSCommand( vmId = 68655de5-7bd9-4bdc-8628-3b2069ec312f), log id: 5672c39a >2012-11-08 07:53:47,369 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (pool-4-thread-48) FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 5672c39a >2012-11-08 07:53:47,403 INFO [org.ovirt.engine.core.bll.VdsSelector] (pool-4-thread-48) VDS rhs-gp-srv11.lab.eng.blr.redhat.com 96739be4-2990-11e2-8577-52540035ed48 is the same host the VM is currently running on VDS rhs-client11.lab.eng.blr.redhat.com 90b581c2-2977-11e2-82cd-52540035ed48 is not in up status or belongs to the VM's cluster VDS rhs-gp-srv4.lab.eng.blr.redhat.com d2bd32d4-2992-11e2-ab59-52540035ed48 is not in up status or belongs to the VM's cluster VDS rhs-arch-srv2.lab.eng.blr.redhat.com cf017816-29a2-11e2-949e-52540035ed48 is not in up status or belongs to the VM's cluster >2012-11-08 07:53:47,403 WARN [org.ovirt.engine.core.bll.RunVmOnceCommand] (pool-4-thread-48) CanDoAction of action RunVmOnce failed. Reasons:VAR__ACTION__RUN,VAR__TYPE__VM,$VmName b2,VAR__ACTION__RUN,VAR__TYPE__VM,ACTION_TYPE_FAILED_VDS_VM_CLUSTER >2012-11-08 07:54:01,584 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-26) Failed to decrypt Data must not be longer than 256 bytes >2012-11-08 07:54:01,584 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-21) Failed to decrypt Data must not be longer than 256 bytes >2012-11-08 07:54:09,628 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-43) Failed to decrypt Data must not be longer than 256 bytes >2012-11-08 07:54:09,628 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-38) Failed to decrypt Data must not be longer than 256 bytes >2012-11-08 07:54:37,828 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-12) Failed to decrypt Data must not be longer than 256 bytes >2012-11-08 07:54:37,828 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-7) Failed to decrypt Data must not be longer than 256 bytes >2012-11-08 07:55:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-63) Autorecovering hosts is disabled, skipping >2012-11-08 07:55:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-63) Autorecovering storage domains is disabled, skipping >2012-11-08 07:56:04,422 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-13) Failed to decrypt Data must not be longer than 256 bytes >2012-11-08 07:56:04,422 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-11) Failed to decrypt Data must not be longer than 256 bytes >2012-11-08 07:56:06,900 INFO [org.ovirt.engine.core.bll.RemoveVdsCommand] (ajp-/127.0.0.1:8702-14) Lock Acquired to object EngineLock [exclusiveLocks= key: 90b581c2-2977-11e2-82cd-52540035ed48 value: VDS >, sharedLocks= ] >2012-11-08 07:56:06,990 INFO [org.ovirt.engine.core.bll.RemoveVdsCommand] (pool-4-thread-50) [33241c3e] Running command: RemoveVdsCommand internal: false. Entities affected : ID: 90b581c2-2977-11e2-82cd-52540035ed48 Type: VDS >2012-11-08 07:56:07,021 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (pool-4-thread-50) [33241c3e] START, RemoveVdsVDSCommand( HostId = 90b581c2-2977-11e2-82cd-52540035ed48), log id: 40be444e >2012-11-08 07:56:07,022 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (pool-4-thread-50) [33241c3e] vdsManager::disposing >2012-11-08 07:56:07,022 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (pool-4-thread-50) [33241c3e] FINISH, RemoveVdsVDSCommand, log id: 40be444e >2012-11-08 07:56:07,023 INFO [org.ovirt.engine.core.bll.RemoveVdsCommand] (pool-4-thread-50) [33241c3e] Lock freed to object EngineLock [exclusiveLocks= key: 90b581c2-2977-11e2-82cd-52540035ed48 value: VDS >, sharedLocks= ] >2012-11-08 07:56:08,436 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-28) Failed to decrypt Data must not be longer than 256 bytes >2012-11-08 07:56:08,436 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-25) Failed to decrypt Data must not be longer than 256 bytes >2012-11-08 08:00:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-54) Autorecovering hosts is disabled, skipping >2012-11-08 08:00:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-54) Autorecovering storage domains is disabled, skipping >2012-11-08 08:05:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-5) Autorecovering hosts is disabled, skipping >2012-11-08 08:05:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-5) Autorecovering storage domains is disabled, skipping >2012-11-08 08:10:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-54) Autorecovering hosts is disabled, skipping >2012-11-08 08:10:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-54) Autorecovering storage domains is disabled, skipping >2012-11-08 08:10:25,267 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsoPrefixVDSCommand] (ajp-/127.0.0.1:8702-11) [4461c16b] START, IsoPrefixVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 46447f74 >2012-11-08 08:10:25,268 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsoPrefixVDSCommand] (ajp-/127.0.0.1:8702-11) [4461c16b] FINISH, IsoPrefixVDSCommand, return: /rhev/data-center/3a17dcac-a6e3-4563-a4b0-f27d89b68dd6/7a268ac7-857a-4c53-966d-247deb509161/images/11111111-1111-1111-1111-111111111111, log id: 46447f74 >2012-11-08 08:10:25,288 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-11) [4461c16b] START, IsValidVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: d1be697 >2012-11-08 08:10:25,289 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-11) [4461c16b] FINISH, IsValidVDSCommand, return: true, log id: d1be697 >2012-11-08 08:10:25,299 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp-/127.0.0.1:8702-11) [4461c16b] START, IsVmDuringInitiatingVDSCommand( vmId = 67bd2d56-8e0b-4328-bcfd-ea7f4758c885), log id: 87da61c >2012-11-08 08:10:25,299 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp-/127.0.0.1:8702-11) [4461c16b] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 87da61c >2012-11-08 08:10:25,336 INFO [org.ovirt.engine.core.bll.RunVmOnceCommand] (ajp-/127.0.0.1:8702-11) [4461c16b] Running command: RunVmOnceCommand internal: false. Entities affected : ID: 67bd2d56-8e0b-4328-bcfd-ea7f4758c885 Type: VM, ID: 67bd2d56-8e0b-4328-bcfd-ea7f4758c885 Type: VM >2012-11-08 08:10:25,401 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-11) [4461c16b] START, CreateVmVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=67bd2d56-8e0b-4328-bcfd-ea7f4758c885, vm=org.ovirt.engine.core.common.businessentities.VM@4a090b59), log id: 1c88cc59 >2012-11-08 08:10:25,414 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8702-11) [4461c16b] START, CreateVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=67bd2d56-8e0b-4328-bcfd-ea7f4758c885, vm=org.ovirt.engine.core.common.businessentities.VM@4a090b59), log id: 521d1465 >2012-11-08 08:10:25,460 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8702-11) [4461c16b] org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand spiceSslCipherSuite=DEFAULT,memSize=512,kvmEnable=true,smp=1,emulatedMachine=rhel6.3.0,vmType=kvm,keyboardLayout=en-us,nice=0,display=qxl,smpCoresPerSocket=1,spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard,timeOffset=-43200,transparentHugePages=true,vmId=67bd2d56-8e0b-4328-bcfd-ea7f4758c885,devices=[Ljava.util.Map;@14a1142c,acpiEnable=true,vmName=b1,cpuType=Nehalem,custom={viodiskcache=none} >2012-11-08 08:10:25,460 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8702-11) [4461c16b] FINISH, CreateVDSCommand, log id: 521d1465 >2012-11-08 08:10:25,461 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-11) [4461c16b] IncreasePendingVms::CreateVmIncreasing vds rhs-gp-srv11.lab.eng.blr.redhat.com pending vcpu count, now 1. Vm: b1 >2012-11-08 08:10:25,463 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-11) [4461c16b] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 1c88cc59 >2012-11-08 08:10:27,196 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (QuartzScheduler_Worker-1) START, DestroyVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=67bd2d56-8e0b-4328-bcfd-ea7f4758c885, force=false, secondsToWait=0, gracefully=false), log id: 54e939c1 >2012-11-08 08:10:27,261 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (QuartzScheduler_Worker-1) FINISH, DestroyVDSCommand, log id: 54e939c1 >2012-11-08 08:10:27,281 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-1) Running on vds during rerun failed vm: null >2012-11-08 08:10:27,282 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-1) vm b1 running in db and not running in vds - add to rerun treatment. vds rhs-gp-srv11.lab.eng.blr.redhat.com >2012-11-08 08:10:27,286 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-1) Rerun vm 67bd2d56-8e0b-4328-bcfd-ea7f4758c885. Called from vds rhs-gp-srv11.lab.eng.blr.redhat.com >2012-11-08 08:10:27,313 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsDynamicDataVDSCommand] (pool-4-thread-50) START, UpdateVdsDynamicDataVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vdsDynamic=org.ovirt.engine.core.common.businessentities.VdsDynamic@dbea1aad), log id: b975742 >2012-11-08 08:10:27,314 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsDynamicDataVDSCommand] (pool-4-thread-50) FINISH, UpdateVdsDynamicDataVDSCommand, log id: b975742 >2012-11-08 08:10:27,364 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (pool-4-thread-50) START, IsValidVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 71856217 >2012-11-08 08:10:27,365 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (pool-4-thread-50) FINISH, IsValidVDSCommand, return: true, log id: 71856217 >2012-11-08 08:10:27,380 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (pool-4-thread-50) START, IsVmDuringInitiatingVDSCommand( vmId = 67bd2d56-8e0b-4328-bcfd-ea7f4758c885), log id: 39db7698 >2012-11-08 08:10:27,380 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (pool-4-thread-50) FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 39db7698 >2012-11-08 08:10:27,414 INFO [org.ovirt.engine.core.bll.VdsSelector] (pool-4-thread-50) VDS rhs-gp-srv11.lab.eng.blr.redhat.com 96739be4-2990-11e2-8577-52540035ed48 is the same host the VM is currently running on VDS rhs-gp-srv4.lab.eng.blr.redhat.com d2bd32d4-2992-11e2-ab59-52540035ed48 is not in up status or belongs to the VM's cluster VDS rhs-arch-srv2.lab.eng.blr.redhat.com cf017816-29a2-11e2-949e-52540035ed48 is not in up status or belongs to the VM's cluster >2012-11-08 08:10:27,414 WARN [org.ovirt.engine.core.bll.RunVmOnceCommand] (pool-4-thread-50) CanDoAction of action RunVmOnce failed. Reasons:VAR__ACTION__RUN,VAR__TYPE__VM,$VmName b1,VAR__ACTION__RUN,VAR__TYPE__VM,ACTION_TYPE_FAILED_VDS_VM_CLUSTER >2012-11-08 08:11:17,993 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsoPrefixVDSCommand] (ajp-/127.0.0.1:8702-1) [3d46e482] START, IsoPrefixVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 8fb5eb7 >2012-11-08 08:11:17,993 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsoPrefixVDSCommand] (ajp-/127.0.0.1:8702-1) [3d46e482] FINISH, IsoPrefixVDSCommand, return: /rhev/data-center/3a17dcac-a6e3-4563-a4b0-f27d89b68dd6/7a268ac7-857a-4c53-966d-247deb509161/images/11111111-1111-1111-1111-111111111111, log id: 8fb5eb7 >2012-11-08 08:11:18,031 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-1) [3d46e482] START, IsValidVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 48a53287 >2012-11-08 08:11:18,032 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-1) [3d46e482] FINISH, IsValidVDSCommand, return: true, log id: 48a53287 >2012-11-08 08:11:18,049 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp-/127.0.0.1:8702-1) [3d46e482] START, IsVmDuringInitiatingVDSCommand( vmId = 67bd2d56-8e0b-4328-bcfd-ea7f4758c885), log id: 1d311881 >2012-11-08 08:11:18,049 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp-/127.0.0.1:8702-1) [3d46e482] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 1d311881 >2012-11-08 08:11:18,095 INFO [org.ovirt.engine.core.bll.RunVmOnceCommand] (ajp-/127.0.0.1:8702-1) [3d46e482] Running command: RunVmOnceCommand internal: false. Entities affected : ID: 67bd2d56-8e0b-4328-bcfd-ea7f4758c885 Type: VM, ID: 67bd2d56-8e0b-4328-bcfd-ea7f4758c885 Type: VM >2012-11-08 08:11:18,161 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-1) [3d46e482] START, CreateVmVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=67bd2d56-8e0b-4328-bcfd-ea7f4758c885, vm=org.ovirt.engine.core.common.businessentities.VM@7ef433fa), log id: fece8c3 >2012-11-08 08:11:18,173 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8702-1) [3d46e482] START, CreateVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=67bd2d56-8e0b-4328-bcfd-ea7f4758c885, vm=org.ovirt.engine.core.common.businessentities.VM@7ef433fa), log id: 79153fab >2012-11-08 08:11:18,213 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8702-1) [3d46e482] org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand spiceSslCipherSuite=DEFAULT,memSize=512,kvmEnable=true,smp=1,emulatedMachine=rhel6.3.0,vmType=kvm,keyboardLayout=en-us,nice=0,display=qxl,smpCoresPerSocket=1,spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard,timeOffset=-43200,transparentHugePages=true,vmId=67bd2d56-8e0b-4328-bcfd-ea7f4758c885,devices=[Ljava.util.Map;@75f7213d,acpiEnable=true,vmName=b1,cpuType=Nehalem,custom={viodiskcache=none} >2012-11-08 08:11:18,213 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8702-1) [3d46e482] FINISH, CreateVDSCommand, log id: 79153fab >2012-11-08 08:11:18,213 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-1) [3d46e482] IncreasePendingVms::CreateVmIncreasing vds rhs-gp-srv11.lab.eng.blr.redhat.com pending vcpu count, now 1. Vm: b1 >2012-11-08 08:11:18,216 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-1) [3d46e482] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: fece8c3 >2012-11-08 08:11:20,471 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (QuartzScheduler_Worker-3) START, DestroyVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=67bd2d56-8e0b-4328-bcfd-ea7f4758c885, force=false, secondsToWait=0, gracefully=false), log id: 100e3930 >2012-11-08 08:11:20,528 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (QuartzScheduler_Worker-3) FINISH, DestroyVDSCommand, log id: 100e3930 >2012-11-08 08:11:20,547 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-3) Running on vds during rerun failed vm: null >2012-11-08 08:11:20,548 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-3) vm b1 running in db and not running in vds - add to rerun treatment. vds rhs-gp-srv11.lab.eng.blr.redhat.com >2012-11-08 08:11:20,552 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-3) Rerun vm 67bd2d56-8e0b-4328-bcfd-ea7f4758c885. Called from vds rhs-gp-srv11.lab.eng.blr.redhat.com >2012-11-08 08:11:20,577 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsDynamicDataVDSCommand] (pool-4-thread-47) START, UpdateVdsDynamicDataVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vdsDynamic=org.ovirt.engine.core.common.businessentities.VdsDynamic@dbea1aad), log id: 3edcdaf6 >2012-11-08 08:11:20,579 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsDynamicDataVDSCommand] (pool-4-thread-47) FINISH, UpdateVdsDynamicDataVDSCommand, log id: 3edcdaf6 >2012-11-08 08:11:20,598 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (pool-4-thread-47) START, IsValidVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 37d11a2b >2012-11-08 08:11:20,599 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (pool-4-thread-47) FINISH, IsValidVDSCommand, return: true, log id: 37d11a2b >2012-11-08 08:11:20,617 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (pool-4-thread-47) START, IsVmDuringInitiatingVDSCommand( vmId = 67bd2d56-8e0b-4328-bcfd-ea7f4758c885), log id: 360b0c22 >2012-11-08 08:11:20,617 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (pool-4-thread-47) FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 360b0c22 >2012-11-08 08:11:20,645 INFO [org.ovirt.engine.core.bll.VdsSelector] (pool-4-thread-47) VDS rhs-gp-srv11.lab.eng.blr.redhat.com 96739be4-2990-11e2-8577-52540035ed48 is the same host the VM is currently running on VDS rhs-gp-srv4.lab.eng.blr.redhat.com d2bd32d4-2992-11e2-ab59-52540035ed48 is not in up status or belongs to the VM's cluster VDS rhs-arch-srv2.lab.eng.blr.redhat.com cf017816-29a2-11e2-949e-52540035ed48 is not in up status or belongs to the VM's cluster >2012-11-08 08:11:20,646 WARN [org.ovirt.engine.core.bll.RunVmOnceCommand] (pool-4-thread-47) CanDoAction of action RunVmOnce failed. Reasons:VAR__ACTION__RUN,VAR__TYPE__VM,$VmName b1,VAR__ACTION__RUN,VAR__TYPE__VM,ACTION_TYPE_FAILED_VDS_VM_CLUSTER >2012-11-08 08:12:09,788 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsoPrefixVDSCommand] (ajp-/127.0.0.1:8702-8) [1df07e93] START, IsoPrefixVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: b547676 >2012-11-08 08:12:09,788 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsoPrefixVDSCommand] (ajp-/127.0.0.1:8702-8) [1df07e93] FINISH, IsoPrefixVDSCommand, return: /rhev/data-center/3a17dcac-a6e3-4563-a4b0-f27d89b68dd6/7a268ac7-857a-4c53-966d-247deb509161/images/11111111-1111-1111-1111-111111111111, log id: b547676 >2012-11-08 08:12:09,813 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-8) [1df07e93] START, IsValidVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3db07b9f >2012-11-08 08:12:09,814 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-8) [1df07e93] FINISH, IsValidVDSCommand, return: true, log id: 3db07b9f >2012-11-08 08:12:09,824 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp-/127.0.0.1:8702-8) [1df07e93] START, IsVmDuringInitiatingVDSCommand( vmId = 67bd2d56-8e0b-4328-bcfd-ea7f4758c885), log id: 52e09fc4 >2012-11-08 08:12:09,824 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp-/127.0.0.1:8702-8) [1df07e93] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 52e09fc4 >2012-11-08 08:12:09,858 INFO [org.ovirt.engine.core.bll.RunVmOnceCommand] (ajp-/127.0.0.1:8702-8) [1df07e93] Running command: RunVmOnceCommand internal: false. Entities affected : ID: 67bd2d56-8e0b-4328-bcfd-ea7f4758c885 Type: VM, ID: 67bd2d56-8e0b-4328-bcfd-ea7f4758c885 Type: VM >2012-11-08 08:12:09,925 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-8) [1df07e93] START, CreateVmVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=67bd2d56-8e0b-4328-bcfd-ea7f4758c885, vm=org.ovirt.engine.core.common.businessentities.VM@52d0d7c6), log id: 5fa7525a >2012-11-08 08:12:09,944 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8702-8) [1df07e93] START, CreateVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=67bd2d56-8e0b-4328-bcfd-ea7f4758c885, vm=org.ovirt.engine.core.common.businessentities.VM@52d0d7c6), log id: 3e704585 >2012-11-08 08:12:09,985 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8702-8) [1df07e93] org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand spiceSslCipherSuite=DEFAULT,memSize=512,kvmEnable=true,smp=1,emulatedMachine=rhel6.3.0,vmType=kvm,keyboardLayout=en-us,nice=0,display=qxl,smpCoresPerSocket=1,spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard,timeOffset=-43200,transparentHugePages=true,vmId=67bd2d56-8e0b-4328-bcfd-ea7f4758c885,devices=[Ljava.util.Map;@290c3c3a,acpiEnable=true,vmName=b1,cpuType=Nehalem,custom={viodiskcache=none} >2012-11-08 08:12:09,985 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8702-8) [1df07e93] FINISH, CreateVDSCommand, log id: 3e704585 >2012-11-08 08:12:09,986 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-8) [1df07e93] IncreasePendingVms::CreateVmIncreasing vds rhs-gp-srv11.lab.eng.blr.redhat.com pending vcpu count, now 1. Vm: b1 >2012-11-08 08:12:09,988 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-8) [1df07e93] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 5fa7525a >2012-11-08 08:12:11,692 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-5) VM b1 67bd2d56-8e0b-4328-bcfd-ea7f4758c885 moved from WaitForLaunch --> PoweringUp >2012-11-08 08:12:11,697 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (QuartzScheduler_Worker-5) START, FullListVdsCommand( HostId = 96739be4-2990-11e2-8577-52540035ed48, vds=null, vmIds=[67bd2d56-8e0b-4328-bcfd-ea7f4758c885]), log id: 7da8f3d5 >2012-11-08 08:12:11,719 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (QuartzScheduler_Worker-5) FINISH, FullListVdsCommand, return: [Lorg.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcStruct;@47e13c54, log id: 7da8f3d5 >2012-11-08 08:12:41,502 INFO [org.ovirt.engine.core.bll.SetVmTicketCommand] (ajp-/127.0.0.1:8702-15) [7e203ba1] Running command: SetVmTicketCommand internal: false. Entities affected : ID: 67bd2d56-8e0b-4328-bcfd-ea7f4758c885 Type: VM >2012-11-08 08:12:41,525 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp-/127.0.0.1:8702-15) [7e203ba1] START, SetVmTicketVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=67bd2d56-8e0b-4328-bcfd-ea7f4758c885, ticket=RYvnvO6FWddR, validTime=120,m userName=admin@internal, userId=fdfc627c-d875-11e0-90f0-83df133b58cc), log id: 1a62bab8 >2012-11-08 08:12:41,554 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp-/127.0.0.1:8702-15) [7e203ba1] FINISH, SetVmTicketVDSCommand, log id: 1a62bab8 >2012-11-08 08:13:11,920 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-17) [36467d8f] VM b1 67bd2d56-8e0b-4328-bcfd-ea7f4758c885 moved from PoweringUp --> Up >2012-11-08 08:13:11,953 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsDynamicDataVDSCommand] (QuartzScheduler_Worker-17) [36467d8f] START, UpdateVdsDynamicDataVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vdsDynamic=org.ovirt.engine.core.common.businessentities.VdsDynamic@765ecb78), log id: 3a3ddf15 >2012-11-08 08:13:11,954 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsDynamicDataVDSCommand] (QuartzScheduler_Worker-17) [36467d8f] FINISH, UpdateVdsDynamicDataVDSCommand, log id: 3a3ddf15 >2012-11-08 08:15:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-14) Autorecovering hosts is disabled, skipping >2012-11-08 08:15:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-14) Autorecovering storage domains is disabled, skipping >2012-11-08 08:15:57,377 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsoPrefixVDSCommand] (ajp-/127.0.0.1:8702-1) [432aee0d] START, IsoPrefixVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 54c07d2b >2012-11-08 08:15:57,377 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsoPrefixVDSCommand] (ajp-/127.0.0.1:8702-1) [432aee0d] FINISH, IsoPrefixVDSCommand, return: /rhev/data-center/3a17dcac-a6e3-4563-a4b0-f27d89b68dd6/7a268ac7-857a-4c53-966d-247deb509161/images/11111111-1111-1111-1111-111111111111, log id: 54c07d2b >2012-11-08 08:15:57,407 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-1) [432aee0d] START, IsValidVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 43392974 >2012-11-08 08:15:57,407 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-1) [432aee0d] FINISH, IsValidVDSCommand, return: true, log id: 43392974 >2012-11-08 08:15:57,419 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp-/127.0.0.1:8702-1) [432aee0d] START, IsVmDuringInitiatingVDSCommand( vmId = 68655de5-7bd9-4bdc-8628-3b2069ec312f), log id: 69b788a >2012-11-08 08:15:57,419 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp-/127.0.0.1:8702-1) [432aee0d] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 69b788a >2012-11-08 08:15:57,451 INFO [org.ovirt.engine.core.bll.RunVmOnceCommand] (ajp-/127.0.0.1:8702-1) [432aee0d] Running command: RunVmOnceCommand internal: false. Entities affected : ID: 68655de5-7bd9-4bdc-8628-3b2069ec312f Type: VM, ID: 68655de5-7bd9-4bdc-8628-3b2069ec312f Type: VM >2012-11-08 08:15:57,524 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-1) [432aee0d] START, CreateVmVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=68655de5-7bd9-4bdc-8628-3b2069ec312f, vm=org.ovirt.engine.core.common.businessentities.VM@3db87041), log id: 74b8a392 >2012-11-08 08:15:57,537 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8702-1) [432aee0d] START, CreateVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=68655de5-7bd9-4bdc-8628-3b2069ec312f, vm=org.ovirt.engine.core.common.businessentities.VM@3db87041), log id: 4c8f564c >2012-11-08 08:15:57,578 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8702-1) [432aee0d] org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand spiceSslCipherSuite=DEFAULT,memSize=512,kvmEnable=true,smp=1,emulatedMachine=rhel6.3.0,vmType=kvm,keyboardLayout=en-us,nice=0,display=qxl,smpCoresPerSocket=1,spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard,timeOffset=-43200,transparentHugePages=true,vmId=68655de5-7bd9-4bdc-8628-3b2069ec312f,devices=[Ljava.util.Map;@425cd184,acpiEnable=true,vmName=b2,cpuType=Nehalem,custom={viodiskcache=writeback} >2012-11-08 08:15:57,578 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8702-1) [432aee0d] FINISH, CreateVDSCommand, log id: 4c8f564c >2012-11-08 08:15:57,579 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-1) [432aee0d] IncreasePendingVms::CreateVmIncreasing vds rhs-gp-srv11.lab.eng.blr.redhat.com pending vcpu count, now 1. Vm: b2 >2012-11-08 08:15:57,581 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-1) [432aee0d] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 74b8a392 >2012-11-08 08:15:59,988 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-23) VM b2 68655de5-7bd9-4bdc-8628-3b2069ec312f moved from WaitForLaunch --> PoweringUp >2012-11-08 08:15:59,988 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (QuartzScheduler_Worker-23) START, FullListVdsCommand( HostId = 96739be4-2990-11e2-8577-52540035ed48, vds=null, vmIds=[68655de5-7bd9-4bdc-8628-3b2069ec312f]), log id: 36d69e60 >2012-11-08 08:16:00,013 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (QuartzScheduler_Worker-23) FINISH, FullListVdsCommand, return: [Lorg.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcStruct;@4437ccc0, log id: 36d69e60 >2012-11-08 08:16:10,577 INFO [org.ovirt.engine.core.bll.SetVmTicketCommand] (ajp-/127.0.0.1:8702-14) [74339311] Running command: SetVmTicketCommand internal: false. Entities affected : ID: 68655de5-7bd9-4bdc-8628-3b2069ec312f Type: VM >2012-11-08 08:16:10,605 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp-/127.0.0.1:8702-14) [74339311] START, SetVmTicketVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=68655de5-7bd9-4bdc-8628-3b2069ec312f, ticket=GFsQLjci1LnC, validTime=120,m userName=admin@internal, userId=fdfc627c-d875-11e0-90f0-83df133b58cc), log id: 44407062 >2012-11-08 08:16:10,647 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp-/127.0.0.1:8702-14) [74339311] FINISH, SetVmTicketVDSCommand, log id: 44407062 >2012-11-08 08:16:38,071 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-5) [49c8438b] START, IsValidVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: fb26fe6 >2012-11-08 08:16:38,072 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-5) [49c8438b] FINISH, IsValidVDSCommand, return: true, log id: fb26fe6 >2012-11-08 08:16:38,081 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp-/127.0.0.1:8702-5) [49c8438b] START, IsVmDuringInitiatingVDSCommand( vmId = 7fa6f92f-755f-44c3-8f35-b4c56410c87b), log id: 110c8c5e >2012-11-08 08:16:38,081 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp-/127.0.0.1:8702-5) [49c8438b] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 110c8c5e >2012-11-08 08:16:38,118 INFO [org.ovirt.engine.core.bll.RunVmOnceCommand] (ajp-/127.0.0.1:8702-5) [49c8438b] Running command: RunVmOnceCommand internal: false. Entities affected : ID: 7fa6f92f-755f-44c3-8f35-b4c56410c87b Type: VM >2012-11-08 08:16:38,163 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsoPrefixVDSCommand] (ajp-/127.0.0.1:8702-5) [49c8438b] START, IsoPrefixVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 5f07aec4 >2012-11-08 08:16:38,163 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsoPrefixVDSCommand] (ajp-/127.0.0.1:8702-5) [49c8438b] FINISH, IsoPrefixVDSCommand, return: /rhev/data-center/3a17dcac-a6e3-4563-a4b0-f27d89b68dd6/7a268ac7-857a-4c53-966d-247deb509161/images/11111111-1111-1111-1111-111111111111, log id: 5f07aec4 >2012-11-08 08:16:38,174 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-5) [49c8438b] START, CreateVmVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=7fa6f92f-755f-44c3-8f35-b4c56410c87b, vm=org.ovirt.engine.core.common.businessentities.VM@75f9d98e), log id: 7162f699 >2012-11-08 08:16:38,187 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8702-5) [49c8438b] START, CreateVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=7fa6f92f-755f-44c3-8f35-b4c56410c87b, vm=org.ovirt.engine.core.common.businessentities.VM@75f9d98e), log id: 32ca1c0e >2012-11-08 08:16:38,228 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8702-5) [49c8438b] org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand spiceSslCipherSuite=DEFAULT,memSize=512,kvmEnable=true,smp=1,emulatedMachine=rhel6.3.0,vmType=kvm,keyboardLayout=en-us,nice=0,display=qxl,smpCoresPerSocket=1,spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard,timeOffset=-43200,transparentHugePages=true,vmId=7fa6f92f-755f-44c3-8f35-b4c56410c87b,devices=[Ljava.util.Map;@31285d29,acpiEnable=true,vmName=b3,cpuType=Nehalem,custom={} >2012-11-08 08:16:38,228 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8702-5) [49c8438b] FINISH, CreateVDSCommand, log id: 32ca1c0e >2012-11-08 08:16:38,228 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-5) [49c8438b] IncreasePendingVms::CreateVmIncreasing vds rhs-gp-srv11.lab.eng.blr.redhat.com pending vcpu count, now 2. Vm: b3 >2012-11-08 08:16:38,230 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-5) [49c8438b] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 7162f699 >2012-11-08 08:16:39,416 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-91) VM b3 7fa6f92f-755f-44c3-8f35-b4c56410c87b moved from WaitForLaunch --> PoweringUp >2012-11-08 08:16:39,417 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (QuartzScheduler_Worker-91) START, FullListVdsCommand( HostId = 96739be4-2990-11e2-8577-52540035ed48, vds=null, vmIds=[7fa6f92f-755f-44c3-8f35-b4c56410c87b]), log id: 568a2ba >2012-11-08 08:16:39,437 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (QuartzScheduler_Worker-91) FINISH, FullListVdsCommand, return: [Lorg.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcStruct;@1893641c, log id: 568a2ba >2012-11-08 08:16:47,623 INFO [org.ovirt.engine.core.bll.SetVmTicketCommand] (ajp-/127.0.0.1:8702-16) [554d8a54] Running command: SetVmTicketCommand internal: false. Entities affected : ID: 7fa6f92f-755f-44c3-8f35-b4c56410c87b Type: VM >2012-11-08 08:16:47,644 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp-/127.0.0.1:8702-16) [554d8a54] START, SetVmTicketVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=7fa6f92f-755f-44c3-8f35-b4c56410c87b, ticket=gPFttScO4q4t, validTime=120,m userName=admin@internal, userId=fdfc627c-d875-11e0-90f0-83df133b58cc), log id: 5317b8d7 >2012-11-08 08:16:47,673 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp-/127.0.0.1:8702-16) [554d8a54] FINISH, SetVmTicketVDSCommand, log id: 5317b8d7 >2012-11-08 08:16:58,288 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-34) VM b2 68655de5-7bd9-4bdc-8628-3b2069ec312f moved from PoweringUp --> Up >2012-11-08 08:16:58,319 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsDynamicDataVDSCommand] (QuartzScheduler_Worker-34) START, UpdateVdsDynamicDataVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vdsDynamic=org.ovirt.engine.core.common.businessentities.VdsDynamic@a5f31317), log id: a8d23d9 >2012-11-08 08:16:58,321 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsDynamicDataVDSCommand] (QuartzScheduler_Worker-34) FINISH, UpdateVdsDynamicDataVDSCommand, log id: a8d23d9 >2012-11-08 08:17:39,842 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-97) VM b3 7fa6f92f-755f-44c3-8f35-b4c56410c87b moved from PoweringUp --> Up >2012-11-08 08:17:39,871 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsDynamicDataVDSCommand] (QuartzScheduler_Worker-97) START, UpdateVdsDynamicDataVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vdsDynamic=org.ovirt.engine.core.common.businessentities.VdsDynamic@54a81ea), log id: 51cec855 >2012-11-08 08:17:39,872 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsDynamicDataVDSCommand] (QuartzScheduler_Worker-97) FINISH, UpdateVdsDynamicDataVDSCommand, log id: 51cec855 >2012-11-08 08:17:58,011 INFO [org.ovirt.engine.core.bll.LoginAdminUserCommand] (ajp-/127.0.0.1:8702-1) Checking if user admin@internal is an admin, result true >2012-11-08 08:17:58,011 INFO [org.ovirt.engine.core.bll.LoginAdminUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginAdminUserCommand internal: false. >2012-11-08 08:19:06,178 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-16) [7dd17529] START, IsValidVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 43c93925 >2012-11-08 08:19:06,179 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-16) [7dd17529] FINISH, IsValidVDSCommand, return: true, log id: 43c93925 >2012-11-08 08:19:06,189 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp-/127.0.0.1:8702-16) [7dd17529] START, IsVmDuringInitiatingVDSCommand( vmId = 661d6af9-9024-4744-91b1-258e38466815), log id: 4d4c795 >2012-11-08 08:19:06,189 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp-/127.0.0.1:8702-16) [7dd17529] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 4d4c795 >2012-11-08 08:19:06,216 INFO [org.ovirt.engine.core.bll.RunVmOnceCommand] (ajp-/127.0.0.1:8702-16) [7dd17529] Running command: RunVmOnceCommand internal: false. Entities affected : ID: 661d6af9-9024-4744-91b1-258e38466815 Type: VM >2012-11-08 08:19:06,267 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsoPrefixVDSCommand] (ajp-/127.0.0.1:8702-16) [7dd17529] START, IsoPrefixVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 2e1239a1 >2012-11-08 08:19:06,267 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsoPrefixVDSCommand] (ajp-/127.0.0.1:8702-16) [7dd17529] FINISH, IsoPrefixVDSCommand, return: /rhev/data-center/3a17dcac-a6e3-4563-a4b0-f27d89b68dd6/7a268ac7-857a-4c53-966d-247deb509161/images/11111111-1111-1111-1111-111111111111, log id: 2e1239a1 >2012-11-08 08:19:06,284 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-16) [7dd17529] START, CreateVmVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=661d6af9-9024-4744-91b1-258e38466815, vm=org.ovirt.engine.core.common.businessentities.VM@50cfee66), log id: 2e8590a8 >2012-11-08 08:19:06,297 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8702-16) [7dd17529] START, CreateVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=661d6af9-9024-4744-91b1-258e38466815, vm=org.ovirt.engine.core.common.businessentities.VM@50cfee66), log id: 7d252971 >2012-11-08 08:19:06,362 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8702-16) [7dd17529] org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand spiceSslCipherSuite=DEFAULT,memSize=512,kvmEnable=true,smp=1,emulatedMachine=rhel6.3.0,vmType=kvm,keyboardLayout=en-us,nice=0,display=qxl,smpCoresPerSocket=1,spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard,timeOffset=-43200,transparentHugePages=true,vmId=661d6af9-9024-4744-91b1-258e38466815,devices=[Ljava.util.Map;@5653a69e,acpiEnable=true,vmName=disrep1,cpuType=Nehalem,custom={} >2012-11-08 08:19:06,362 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8702-16) [7dd17529] FINISH, CreateVDSCommand, log id: 7d252971 >2012-11-08 08:19:06,362 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-16) [7dd17529] IncreasePendingVms::CreateVmIncreasing vds rhs-gp-srv11.lab.eng.blr.redhat.com pending vcpu count, now 1. Vm: disrep1 >2012-11-08 08:19:06,364 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-16) [7dd17529] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 2e8590a8 >2012-11-08 08:19:08,160 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-68) VM disrep1 661d6af9-9024-4744-91b1-258e38466815 moved from WaitForLaunch --> PoweringUp >2012-11-08 08:19:08,165 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (QuartzScheduler_Worker-68) START, FullListVdsCommand( HostId = 96739be4-2990-11e2-8577-52540035ed48, vds=null, vmIds=[661d6af9-9024-4744-91b1-258e38466815]), log id: 67a66f0 >2012-11-08 08:19:08,190 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (QuartzScheduler_Worker-68) FINISH, FullListVdsCommand, return: [Lorg.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcStruct;@4dfc19e2, log id: 67a66f0 >2012-11-08 08:19:14,158 INFO [org.ovirt.engine.core.bll.SetVmTicketCommand] (ajp-/127.0.0.1:8702-5) [6ffd7483] Running command: SetVmTicketCommand internal: false. Entities affected : ID: 661d6af9-9024-4744-91b1-258e38466815 Type: VM >2012-11-08 08:19:14,179 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp-/127.0.0.1:8702-5) [6ffd7483] START, SetVmTicketVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=661d6af9-9024-4744-91b1-258e38466815, ticket=K3Xk2iz9PuJ6, validTime=120,m userName=admin@internal, userId=fdfc627c-d875-11e0-90f0-83df133b58cc), log id: 6b0823f5 >2012-11-08 08:19:14,209 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp-/127.0.0.1:8702-5) [6ffd7483] FINISH, SetVmTicketVDSCommand, log id: 6b0823f5 >2012-11-08 08:19:22,904 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (ajp-/127.0.0.1:8702-4) [279dedd] START, ValidateStorageServerConnectionVDSCommand(HostName = rhs-arch-srv2.lab.eng.blr.redhat.com, HostId = cf017816-29a2-11e2-949e-52540035ed48, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = POSIXFS, connectionList = [{ id: null, connection: rhs-client11.lab.eng.blr.redhat.com:/distrep, iqn: null, vfsType: glusterfs, mountOptions: backupvolfile-server=rhs-client12.lab.eng.blr.redhat.com,background-qlen=128,fetch-attempts=3, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 11c1d0fb >2012-11-08 08:19:22,930 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (ajp-/127.0.0.1:8702-4) [279dedd] FINISH, ValidateStorageServerConnectionVDSCommand, return: {00000000-0000-0000-0000-000000000000=0}, log id: 11c1d0fb >2012-11-08 08:19:22,933 INFO [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-4) [279dedd] Running command: AddStorageServerConnectionCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2012-11-08 08:19:22,936 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-4) [279dedd] START, ConnectStorageServerVDSCommand(HostName = rhs-arch-srv2.lab.eng.blr.redhat.com, HostId = cf017816-29a2-11e2-949e-52540035ed48, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = POSIXFS, connectionList = [{ id: null, connection: rhs-client11.lab.eng.blr.redhat.com:/distrep, iqn: null, vfsType: glusterfs, mountOptions: backupvolfile-server=rhs-client12.lab.eng.blr.redhat.com,background-qlen=128,fetch-attempts=3, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 65ec6136 >2012-11-08 08:19:27,164 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-4) [279dedd] FINISH, ConnectStorageServerVDSCommand, return: {00000000-0000-0000-0000-000000000000=0}, log id: 65ec6136 >2012-11-08 08:19:27,198 WARN [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (ajp-/127.0.0.1:8702-13) [7185b551] The message key AddPosixFsStorageDomain is missing from bundles/ExecutionMessages >2012-11-08 08:19:27,214 INFO [org.ovirt.engine.core.bll.storage.AddPosixFsStorageDomainCommand] (ajp-/127.0.0.1:8702-13) [7185b551] Running command: AddPosixFsStorageDomainCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2012-11-08 08:19:27,219 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-13) [7185b551] START, CreateStorageDomainVDSCommand(HostName = rhs-arch-srv2.lab.eng.blr.redhat.com, HostId = cf017816-29a2-11e2-949e-52540035ed48, storageDomain=org.ovirt.engine.core.common.businessentities.storage_domain_static@a975e6bd, args=rhs-client11.lab.eng.blr.redhat.com:/distrep), log id: 332fadf0 >2012-11-08 08:19:30,295 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-13) [7185b551] FINISH, CreateStorageDomainVDSCommand, log id: 332fadf0 >2012-11-08 08:19:30,296 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStorageDomainStatsVDSCommand] (ajp-/127.0.0.1:8702-13) [7185b551] START, GetStorageDomainStatsVDSCommand(HostName = rhs-arch-srv2.lab.eng.blr.redhat.com, HostId = cf017816-29a2-11e2-949e-52540035ed48, storageDomainId=00068b9a-c8e3-4e01-b2c4-963931bfae65), log id: 1c6be53e >2012-11-08 08:19:30,326 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStorageDomainStatsVDSCommand] (ajp-/127.0.0.1:8702-13) [7185b551] FINISH, GetStorageDomainStatsVDSCommand, return: org.ovirt.engine.core.common.businessentities.storage_domains@7ccb0181, log id: 1c6be53e >2012-11-08 08:19:30,450 INFO [org.ovirt.engine.core.bll.storage.AttachStorageDomainToPoolCommand] (ajp-/127.0.0.1:8702-8) [6de25604] Running command: AttachStorageDomainToPoolCommand internal: false. Entities affected : ID: 00068b9a-c8e3-4e01-b2c4-963931bfae65 Type: Storage >2012-11-08 08:19:30,452 INFO [org.ovirt.engine.core.bll.storage.AddStoragePoolWithStoragesCommand] (ajp-/127.0.0.1:8702-8) [10716f7a] Lock Acquired to object EngineLock [exclusiveLocks= key: f4c98a0b-0578-485f-90f4-743df3d05d7e value: POOL >, sharedLocks= ] >2012-11-08 08:19:30,471 INFO [org.ovirt.engine.core.bll.storage.AddStoragePoolWithStoragesCommand] (ajp-/127.0.0.1:8702-8) [10716f7a] Running command: AddStoragePoolWithStoragesCommand internal: true. Entities affected : ID: f4c98a0b-0578-485f-90f4-743df3d05d7e Type: StoragePool >2012-11-08 08:19:30,520 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (ajp-/127.0.0.1:8702-8) [61f6479e] START, ValidateStorageServerConnectionVDSCommand(HostName = rhs-arch-srv2.lab.eng.blr.redhat.com, HostId = cf017816-29a2-11e2-949e-52540035ed48, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = POSIXFS, connectionList = [{ id: 9feeea60-7906-49d7-9e22-6cf07420fc26, connection: rhs-client11.lab.eng.blr.redhat.com:/distrep, iqn: null, vfsType: glusterfs, mountOptions: backupvolfile-server=rhs-client12.lab.eng.blr.redhat.com,background-qlen=128,fetch-attempts=3, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 6fe5e857 >2012-11-08 08:19:30,547 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (ajp-/127.0.0.1:8702-8) [61f6479e] FINISH, ValidateStorageServerConnectionVDSCommand, return: {9feeea60-7906-49d7-9e22-6cf07420fc26=0}, log id: 6fe5e857 >2012-11-08 08:19:30,547 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (ajp-/127.0.0.1:8702-8) [61f6479e] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2012-11-08 08:19:30,548 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-8) [61f6479e] START, ConnectStorageServerVDSCommand(HostName = rhs-arch-srv2.lab.eng.blr.redhat.com, HostId = cf017816-29a2-11e2-949e-52540035ed48, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = POSIXFS, connectionList = [{ id: 9feeea60-7906-49d7-9e22-6cf07420fc26, connection: rhs-client11.lab.eng.blr.redhat.com:/distrep, iqn: null, vfsType: glusterfs, mountOptions: backupvolfile-server=rhs-client12.lab.eng.blr.redhat.com,background-qlen=128,fetch-attempts=3, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 510e886c >2012-11-08 08:19:30,588 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-8) [61f6479e] FINISH, ConnectStorageServerVDSCommand, return: {9feeea60-7906-49d7-9e22-6cf07420fc26=0}, log id: 510e886c >2012-11-08 08:19:30,590 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStoragePoolVDSCommand] (ajp-/127.0.0.1:8702-8) [61f6479e] START, CreateStoragePoolVDSCommand(HostName = rhs-arch-srv2.lab.eng.blr.redhat.com, HostId = cf017816-29a2-11e2-949e-52540035ed48, storagePoolId=f4c98a0b-0578-485f-90f4-743df3d05d7e, storageType=POSIXFS, storagePoolName=RHS-FUNCTIONAL, masterDomainId=00068b9a-c8e3-4e01-b2c4-963931bfae65, domainsIdList=[00068b9a-c8e3-4e01-b2c4-963931bfae65], masterVersion=1), log id: 45feff7a >2012-11-08 08:19:34,546 INFO [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (ajp-/127.0.0.1:8702-2) SSH execute 10.70.36.16 'IDFILE=/etc/vdsm/vdsm.id; if [ -r "${IDFILE}" ]; then cat "${IDFILE}"; else UUID="$(dmidecode -s system-uuid 2> /dev/null | sed -e 's/.*Not.*//' )"; if [ -z "${UUID}" ]; then UUID="$(uuidgen 2> /dev/null)" && mkdir -p "$(dirname "${IDFILE}")" && echo "${UUID}" > "${IDFILE}" && chmod 0644 "${IDFILE}"; fi; [ -n "${UUID}" ] && echo "${UUID}"; fi' >2012-11-08 08:19:34,700 INFO [org.ovirt.engine.core.bll.AddVdsCommand] (ajp-/127.0.0.1:8702-2) [590e46c7] Running command: AddVdsCommand internal: false. Entities affected : ID: c04832b4-2976-11e2-b6fb-52540035ed48 Type: VdsGroups >2012-11-08 08:19:34,715 INFO [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp-/127.0.0.1:8702-2) [3379a55c] Running command: AddVdsSpmIdCommand internal: true. Entities affected : ID: f047378c-29a6-11e2-8a79-52540035ed48 Type: VDS >2012-11-08 08:19:34,719 INFO [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp-/127.0.0.1:8702-2) [3379a55c] Lock freed to object EngineLock [exclusiveLocks= key: 1eaedbbb-1c0e-4028-9b0e-adac7e07d25f value: REGISTER_VDS >, sharedLocks= ] >2012-11-08 08:19:34,719 ERROR [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp-/127.0.0.1:8702-2) [3379a55c] Cannot get vdsManager for vdsid=f047378c-29a6-11e2-8a79-52540035ed48 >2012-11-08 08:19:34,721 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (ajp-/127.0.0.1:8702-2) [3379a55c] START, RemoveVdsVDSCommand(HostName = rhs-gp-srv12.lab.eng.blr.redhat.com, HostId = f047378c-29a6-11e2-8a79-52540035ed48), log id: 2ebfc6a4 >2012-11-08 08:19:34,721 ERROR [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp-/127.0.0.1:8702-2) [3379a55c] Cannot get vdsManager for vdsid=f047378c-29a6-11e2-8a79-52540035ed48 >2012-11-08 08:19:34,721 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (ajp-/127.0.0.1:8702-2) [3379a55c] FINISH, RemoveVdsVDSCommand, log id: 2ebfc6a4 >2012-11-08 08:19:34,721 ERROR [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp-/127.0.0.1:8702-2) [3379a55c] Cannot get vdsManager for vdsid=f047378c-29a6-11e2-8a79-52540035ed48 >2012-11-08 08:19:34,722 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-2) [3379a55c] START, AddVdsVDSCommand(HostName = rhs-gp-srv12.lab.eng.blr.redhat.com, HostId = f047378c-29a6-11e2-8a79-52540035ed48), log id: 582b4c8a >2012-11-08 08:19:34,722 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-2) [3379a55c] AddVds - entered , starting logic to add VDS f047378c-29a6-11e2-8a79-52540035ed48 >2012-11-08 08:19:34,724 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-2) [3379a55c] AddVds - VDS f047378c-29a6-11e2-8a79-52540035ed48 was added, will try to add it to the resource manager >2012-11-08 08:19:34,725 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ajp-/127.0.0.1:8702-2) [3379a55c] Eneterd VdsManager:constructor >2012-11-08 08:19:34,725 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ajp-/127.0.0.1:8702-2) [3379a55c] vdsBroker(10.70.36.16,54,321) >2012-11-08 08:19:34,726 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp-/127.0.0.1:8702-2) [3379a55c] ResourceManager::AddVds - VDS f047378c-29a6-11e2-8a79-52540035ed48 was added to the Resource Manager >2012-11-08 08:19:34,726 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-2) [3379a55c] FINISH, AddVdsVDSCommand, log id: 582b4c8a >2012-11-08 08:19:34,742 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-4-thread-47) [225e1867] Running command: InstallVdsCommand internal: true. Entities affected : ID: f047378c-29a6-11e2-8a79-52540035ed48 Type: VDS >2012-11-08 08:19:34,744 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-4-thread-47) [225e1867] Before Installation pool-4-thread-47 >2012-11-08 08:19:34,744 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-47) [225e1867] Installation of 10.70.36.16. Executing installation stage. (Stage: Starting Host installation) >2012-11-08 08:19:34,744 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-47) [225e1867] Installation of 10.70.36.16. Executing installation stage. (Stage: Connecting to Host) >2012-11-08 08:19:34,809 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-47) [225e1867] Installation of 10.70.36.16. Received message: <BSTRAP component='RHEV_INSTALL' status='OK' message='Connected to Host 10.70.36.16 with SSH key fingerprint: d8:e6:4c:57:0c:54:e1:5a:31:d9:fa:80:44:2a:ac:7f'/>. FYI. (Stage: Connecting to Host) >2012-11-08 08:19:34,837 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-47) [225e1867] Installation of 10.70.36.16. Successfully connected to server ssh. (Stage: Connecting to Host) >2012-11-08 08:19:34,837 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-47) [225e1867] Installation of 10.70.36.16. Executing installation stage. (Stage: Get the unique vds id) >2012-11-08 08:19:34,837 INFO [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-4-thread-47) SSH execute 10.70.36.16 'IDFILE=/etc/vdsm/vdsm.id; if [ -r "${IDFILE}" ]; then cat "${IDFILE}"; else UUID="$(dmidecode -s system-uuid 2> /dev/null | sed -e 's/.*Not.*//' )"; if [ -z "${UUID}" ]; then UUID="$(uuidgen 2> /dev/null)" && mkdir -p "$(dirname "${IDFILE}")" && echo "${UUID}" > "${IDFILE}" && chmod 0644 "${IDFILE}"; fi; [ -n "${UUID}" ] && echo "${UUID}"; fi' >2012-11-08 08:19:34,958 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-61) Installation of 10.70.36.16. Received message: 00000000-0000-0000-0000-0025902F723C. FYI. (Stage: Get the unique vds id) >2012-11-08 08:19:34,970 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-61) Installation of 10.70.36.16. Assigning unique id 00000000-0000-0000-0000-0025902F723C to Host. (Stage: Get the unique vds id) >2012-11-08 08:19:34,974 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-47) [225e1867] Installation of 10.70.36.16. Executing installation stage. (Stage: Upload Installation script to Host) >2012-11-08 08:19:34,974 INFO [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-4-thread-47) SSH execute 10.70.36.16 'cat > '/tmp/ovirt-id_rsa_89746fd4-ead3-4285-a67a-f7a25c5d9799'' >2012-11-08 08:19:35,060 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-47) [225e1867] Installation of 10.70.36.16. Executing installation stage. (Stage: Running first installation script on Host) >2012-11-08 08:19:35,060 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-47) [225e1867] Installation of 10.70.36.16. Sending SSH Command umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar -C "${MYTMP}" --no-same-permissions -o -x && "${MYTMP}"/setup -c 'ssl=true;management_port=54321' -O 'lab.eng.blr.redhat.com' -t 2012-11-08T13:19:35 -S /tmp/ovirt-id_rsa_89746fd4-ead3-4285-a67a-f7a25c5d9799 -p 80 -b -B rhevm http://guido.lab.eng.blr.redhat.com:80/Components/vds/ http://guido.lab.eng.blr.redhat.com:80/Components/vds/ 10.70.36.16 89746fd4-ead3-4285-a67a-f7a25c5d9799 False < /var/cache/ovirt-engine/vdsm-bootstrap-2.tar. (Stage: Running first installation script on Host) >2012-11-08 08:19:35,061 INFO [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-4-thread-47) SSH execute 10.70.36.16 'umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar -C "${MYTMP}" --no-same-permissions -o -x && "${MYTMP}"/setup -c 'ssl=true;management_port=54321' -O 'lab.eng.blr.redhat.com' -t 2012-11-08T13:19:35 -S /tmp/ovirt-id_rsa_89746fd4-ead3-4285-a67a-f7a25c5d9799 -p 80 -b -B rhevm http://guido.lab.eng.blr.redhat.com:80/Components/vds/ http://guido.lab.eng.blr.redhat.com:80/Components/vds/ 10.70.36.16 89746fd4-ead3-4285-a67a-f7a25c5d9799 False' >2012-11-08 08:19:35,165 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-61) Installation of 10.70.36.16. Received message: <BSTRAP component='INSTALLER' status='OK' message='Test platform succeeded'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:19:36,727 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-13) Initializing Host: rhs-gp-srv12.lab.eng.blr.redhat.com >2012-11-08 08:19:37,272 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-61) Installation of 10.70.36.16. Received message: <BSTRAP component='SET_SYSTEM_TIME' status='OK' message='setSystemTime ended successfully'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:19:40,612 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-61) Installation of 10.70.36.16. Received message: <BSTRAP component='RHN_REGISTRATION' status='OK' message='Host properly registered with RHN/Satellite.'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:19:40,782 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-61) Installation of 10.70.36.16. Received message: <BSTRAP component='VDSM_MAJOR_VER' status='OK' message='Available VDSM matches requirements'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:19:40,796 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-61) Installation of 10.70.36.16. Received message: <BSTRAP component='VT_SVM' status='OK' processor='Intel' message='Server supports virtualization'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:19:40,800 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-61) Installation of 10.70.36.16. Received message: <BSTRAP component='OS' status='OK' type='RHEL6' message='Supported platform version'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:19:40,804 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-61) Installation of 10.70.36.16. Received message: <BSTRAP component='KERNEL' status='OK' version='279' message='Supported kernel version: 279'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:19:40,842 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-61) Installation of 10.70.36.16. Received message: <BSTRAP component='REQ PACKAGES' status='WARN' result='SDL.x86_64' message='package SDL.x86_64 is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:19:40,856 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-61) Installation of 10.70.36.16. Received message: <BSTRAP component='REQ PACKAGES' status='WARN' result='bridge-utils.x86_64' message='package bridge-utils.x86_64 is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:19:40,869 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-61) Installation of 10.70.36.16. Received message: <BSTRAP component='REQ PACKAGES' status='WARN' result='mesa-libGLU.x86_64' message='package mesa-libGLU.x86_64 is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:19:40,884 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-61) Installation of 10.70.36.16. Received message: <BSTRAP component='REQ PACKAGES' status='OK' result='openssl.x86_64' message='openssl-1.0.0-25.el6_3.1.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:19:40,899 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-61) Installation of 10.70.36.16. Received message: <BSTRAP component='REQ PACKAGES' status='OK' result='m2crypto.x86_64' message='m2crypto-0.20.2-9.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:19:40,912 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-61) Installation of 10.70.36.16. Received message: <BSTRAP component='REQ PACKAGES' status='WARN' result='rsync.x86_64' message='package rsync.x86_64 is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:19:40,925 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-61) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='WARN' result='qemu-kvm' message='package qemu-kvm is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:19:40,938 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-61) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='WARN' result='qemu-kvm-tools' message='package qemu-kvm-tools is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:19:40,951 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-61) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='WARN' result='(~vdsm~, ~vdsm-cli~)' message='package vdsm is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:19:40,964 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-61) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='WARN' result='libjpeg' message='package libjpeg is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:19:40,977 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-61) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='WARN' result='spice-server' message='package spice-server is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:19:40,990 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-61) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='WARN' result='pixman' message='package pixman is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:19:41,003 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-61) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='WARN' result='seabios' message='package seabios is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:19:41,016 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-61) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='WARN' result='qemu-img' message='package qemu-img is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:19:41,029 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-61) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='WARN' result='fence-agents' message='package fence-agents is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:19:41,043 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-61) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='WARN' result='libselinux-python' message='package libselinux-python is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:19:41,056 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-61) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='WARN' result='sanlock' message='package sanlock is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:19:41,069 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-61) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='WARN' result='sanlock-python' message='package sanlock-python is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:19:46,176 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-1) [6c48debd] START, IsValidVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 106ee928 >2012-11-08 08:19:46,177 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-1) [6c48debd] FINISH, IsValidVDSCommand, return: true, log id: 106ee928 >2012-11-08 08:19:46,190 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp-/127.0.0.1:8702-1) [6c48debd] START, IsVmDuringInitiatingVDSCommand( vmId = 8a0d2e26-440c-4d6c-91d3-1fc4d9b75990), log id: 1a3c1dec >2012-11-08 08:19:46,190 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp-/127.0.0.1:8702-1) [6c48debd] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 1a3c1dec >2012-11-08 08:19:46,239 INFO [org.ovirt.engine.core.bll.RunVmOnceCommand] (ajp-/127.0.0.1:8702-1) [6c48debd] Running command: RunVmOnceCommand internal: false. Entities affected : ID: 8a0d2e26-440c-4d6c-91d3-1fc4d9b75990 Type: VM >2012-11-08 08:19:46,285 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsoPrefixVDSCommand] (ajp-/127.0.0.1:8702-1) [6c48debd] START, IsoPrefixVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 7ac2bfcf >2012-11-08 08:19:46,286 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsoPrefixVDSCommand] (ajp-/127.0.0.1:8702-1) [6c48debd] FINISH, IsoPrefixVDSCommand, return: /rhev/data-center/3a17dcac-a6e3-4563-a4b0-f27d89b68dd6/7a268ac7-857a-4c53-966d-247deb509161/images/11111111-1111-1111-1111-111111111111, log id: 7ac2bfcf >2012-11-08 08:19:46,295 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-1) [6c48debd] START, CreateVmVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=8a0d2e26-440c-4d6c-91d3-1fc4d9b75990, vm=org.ovirt.engine.core.common.businessentities.VM@2088491), log id: 7beee00 >2012-11-08 08:19:46,309 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8702-1) [6c48debd] START, CreateVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=8a0d2e26-440c-4d6c-91d3-1fc4d9b75990, vm=org.ovirt.engine.core.common.businessentities.VM@2088491), log id: 5b172f1 >2012-11-08 08:19:46,358 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8702-1) [6c48debd] org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand spiceSslCipherSuite=DEFAULT,memSize=512,kvmEnable=true,smp=1,emulatedMachine=rhel6.3.0,vmType=kvm,keyboardLayout=en-us,nice=0,display=qxl,smpCoresPerSocket=1,spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard,timeOffset=-43200,transparentHugePages=true,vmId=8a0d2e26-440c-4d6c-91d3-1fc4d9b75990,devices=[Ljava.util.Map;@1b0ff02a,acpiEnable=true,vmName=disrep2,cpuType=Nehalem,custom={} >2012-11-08 08:19:46,358 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8702-1) [6c48debd] FINISH, CreateVDSCommand, log id: 5b172f1 >2012-11-08 08:19:46,358 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-1) [6c48debd] IncreasePendingVms::CreateVmIncreasing vds rhs-gp-srv11.lab.eng.blr.redhat.com pending vcpu count, now 2. Vm: disrep2 >2012-11-08 08:19:46,361 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-1) [6c48debd] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 7beee00 >2012-11-08 08:19:47,695 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-47) VM disrep2 8a0d2e26-440c-4d6c-91d3-1fc4d9b75990 moved from WaitForLaunch --> PoweringUp >2012-11-08 08:19:47,695 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (QuartzScheduler_Worker-47) START, FullListVdsCommand( HostId = 96739be4-2990-11e2-8577-52540035ed48, vds=null, vmIds=[8a0d2e26-440c-4d6c-91d3-1fc4d9b75990]), log id: 192544a1 >2012-11-08 08:19:47,718 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (QuartzScheduler_Worker-47) FINISH, FullListVdsCommand, return: [Lorg.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcStruct;@5dbcb2ec, log id: 192544a1 >2012-11-08 08:19:53,572 INFO [org.ovirt.engine.core.bll.SetVmTicketCommand] (ajp-/127.0.0.1:8702-10) [75a879c5] Running command: SetVmTicketCommand internal: false. Entities affected : ID: 8a0d2e26-440c-4d6c-91d3-1fc4d9b75990 Type: VM >2012-11-08 08:19:53,602 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp-/127.0.0.1:8702-10) [75a879c5] START, SetVmTicketVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=8a0d2e26-440c-4d6c-91d3-1fc4d9b75990, ticket=2KrxgfChks/A, validTime=120,m userName=admin@internal, userId=fdfc627c-d875-11e0-90f0-83df133b58cc), log id: 79721753 >2012-11-08 08:19:53,631 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp-/127.0.0.1:8702-10) [75a879c5] FINISH, SetVmTicketVDSCommand, log id: 79721753 >2012-11-08 08:19:53,945 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStoragePoolVDSCommand] (ajp-/127.0.0.1:8702-8) [61f6479e] FINISH, CreateStoragePoolVDSCommand, log id: 45feff7a >2012-11-08 08:19:53,947 INFO [org.ovirt.engine.core.bll.storage.AddStoragePoolWithStoragesCommand] (ajp-/127.0.0.1:8702-8) [61f6479e] Lock freed to object EngineLock [exclusiveLocks= key: f4c98a0b-0578-485f-90f4-743df3d05d7e value: POOL >, sharedLocks= ] >2012-11-08 08:19:53,948 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp-/127.0.0.1:8702-8) [72b50e23] Lock Acquired to object EngineLock [exclusiveLocks= key: 00068b9a-c8e3-4e01-b2c4-963931bfae65 value: STORAGE >, sharedLocks= ] >2012-11-08 08:19:53,952 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp-/127.0.0.1:8702-8) [72b50e23] Running command: ActivateStorageDomainCommand internal: true. Entities affected : ID: 00068b9a-c8e3-4e01-b2c4-963931bfae65 Type: Storage >2012-11-08 08:19:53,964 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp-/127.0.0.1:8702-8) [72b50e23] Lock freed to object EngineLock [exclusiveLocks= key: 00068b9a-c8e3-4e01-b2c4-963931bfae65 value: STORAGE >, sharedLocks= ] >2012-11-08 08:19:53,965 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp-/127.0.0.1:8702-8) [72b50e23] ActivateStorage Domain. Before Connect all hosts to pool. Time:11/8/12 8:19 AM >2012-11-08 08:19:53,982 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (pool-4-thread-49) [dbefda2] START, ValidateStorageServerConnectionVDSCommand(HostName = rhs-arch-srv2.lab.eng.blr.redhat.com, HostId = cf017816-29a2-11e2-949e-52540035ed48, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = POSIXFS, connectionList = [{ id: 9feeea60-7906-49d7-9e22-6cf07420fc26, connection: rhs-client11.lab.eng.blr.redhat.com:/distrep, iqn: null, vfsType: glusterfs, mountOptions: backupvolfile-server=rhs-client12.lab.eng.blr.redhat.com,background-qlen=128,fetch-attempts=3, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 6330527e >2012-11-08 08:19:54,001 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (pool-4-thread-49) [dbefda2] FINISH, ValidateStorageServerConnectionVDSCommand, return: {9feeea60-7906-49d7-9e22-6cf07420fc26=0}, log id: 6330527e >2012-11-08 08:19:54,001 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (pool-4-thread-49) [dbefda2] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2012-11-08 08:19:54,003 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-4-thread-49) [dbefda2] START, ConnectStorageServerVDSCommand(HostName = rhs-arch-srv2.lab.eng.blr.redhat.com, HostId = cf017816-29a2-11e2-949e-52540035ed48, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = POSIXFS, connectionList = [{ id: 9feeea60-7906-49d7-9e22-6cf07420fc26, connection: rhs-client11.lab.eng.blr.redhat.com:/distrep, iqn: null, vfsType: glusterfs, mountOptions: backupvolfile-server=rhs-client12.lab.eng.blr.redhat.com,background-qlen=128,fetch-attempts=3, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 39c5a2c3 >2012-11-08 08:19:54,035 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-4-thread-49) [dbefda2] FINISH, ConnectStorageServerVDSCommand, return: {9feeea60-7906-49d7-9e22-6cf07420fc26=0}, log id: 39c5a2c3 >2012-11-08 08:19:54,036 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-8) [72b50e23] START, ActivateStorageDomainVDSCommand( storagePoolId = f4c98a0b-0578-485f-90f4-743df3d05d7e, ignoreFailoverLimit = false, compatabilityVersion = null, storageDomainId = 00068b9a-c8e3-4e01-b2c4-963931bfae65), log id: be646ee >2012-11-08 08:19:54,052 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (ajp-/127.0.0.1:8702-8) [4efb64d4] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: f4c98a0b-0578-485f-90f4-743df3d05d7e Type: StoragePool >2012-11-08 08:19:54,089 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (ajp-/127.0.0.1:8702-8) [4efb64d4] hostFromVds::selectedVds - rhs-arch-srv2.lab.eng.blr.redhat.com, spmStatus Unknown_Pool, storage pool RHS-FUNCTIONAL >2012-11-08 08:19:54,097 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (ajp-/127.0.0.1:8702-8) [4efb64d4] START, ConnectStoragePoolVDSCommand(HostName = rhs-arch-srv2.lab.eng.blr.redhat.com, HostId = cf017816-29a2-11e2-949e-52540035ed48, storagePoolId = f4c98a0b-0578-485f-90f4-743df3d05d7e, vds_spm_id = 2, masterDomainId = 00068b9a-c8e3-4e01-b2c4-963931bfae65, masterVersion = 1), log id: 31b53a6d >2012-11-08 08:19:55,302 INFO [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (ajp-/127.0.0.1:8702-15) SSH execute 10.70.36.19 'IDFILE=/etc/vdsm/vdsm.id; if [ -r "${IDFILE}" ]; then cat "${IDFILE}"; else UUID="$(dmidecode -s system-uuid 2> /dev/null | sed -e 's/.*Not.*//' )"; if [ -z "${UUID}" ]; then UUID="$(uuidgen 2> /dev/null)" && mkdir -p "$(dirname "${IDFILE}")" && echo "${UUID}" > "${IDFILE}" && chmod 0644 "${IDFILE}"; fi; [ -n "${UUID}" ] && echo "${UUID}"; fi' >2012-11-08 08:19:55,455 INFO [org.ovirt.engine.core.bll.AddVdsCommand] (ajp-/127.0.0.1:8702-15) [7d4be45f] Running command: AddVdsCommand internal: false. Entities affected : ID: c04832b4-2976-11e2-b6fb-52540035ed48 Type: VdsGroups >2012-11-08 08:19:55,470 INFO [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp-/127.0.0.1:8702-15) [e978fa0] Running command: AddVdsSpmIdCommand internal: true. Entities affected : ID: fca61124-29a6-11e2-a257-52540035ed48 Type: VDS >2012-11-08 08:19:55,473 INFO [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp-/127.0.0.1:8702-15) [e978fa0] Lock freed to object EngineLock [exclusiveLocks= key: 1eaedbbb-1c0e-4028-9b0e-adac7e07d25f value: REGISTER_VDS >, sharedLocks= ] >2012-11-08 08:19:55,474 ERROR [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp-/127.0.0.1:8702-15) [e978fa0] Cannot get vdsManager for vdsid=fca61124-29a6-11e2-a257-52540035ed48 >2012-11-08 08:19:55,475 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (ajp-/127.0.0.1:8702-15) [e978fa0] START, RemoveVdsVDSCommand(HostName = rhs-gp-srv15.lab.eng.blr.redhat.com, HostId = fca61124-29a6-11e2-a257-52540035ed48), log id: 6adc6cc6 >2012-11-08 08:19:55,475 ERROR [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp-/127.0.0.1:8702-15) [e978fa0] Cannot get vdsManager for vdsid=fca61124-29a6-11e2-a257-52540035ed48 >2012-11-08 08:19:55,475 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (ajp-/127.0.0.1:8702-15) [e978fa0] FINISH, RemoveVdsVDSCommand, log id: 6adc6cc6 >2012-11-08 08:19:55,475 ERROR [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp-/127.0.0.1:8702-15) [e978fa0] Cannot get vdsManager for vdsid=fca61124-29a6-11e2-a257-52540035ed48 >2012-11-08 08:19:55,476 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-15) [e978fa0] START, AddVdsVDSCommand(HostName = rhs-gp-srv15.lab.eng.blr.redhat.com, HostId = fca61124-29a6-11e2-a257-52540035ed48), log id: 5244279f >2012-11-08 08:19:55,477 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-15) [e978fa0] AddVds - entered , starting logic to add VDS fca61124-29a6-11e2-a257-52540035ed48 >2012-11-08 08:19:55,479 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-15) [e978fa0] AddVds - VDS fca61124-29a6-11e2-a257-52540035ed48 was added, will try to add it to the resource manager >2012-11-08 08:19:55,479 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ajp-/127.0.0.1:8702-15) [e978fa0] Eneterd VdsManager:constructor >2012-11-08 08:19:55,480 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ajp-/127.0.0.1:8702-15) [e978fa0] vdsBroker(10.70.36.19,54,321) >2012-11-08 08:19:55,481 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp-/127.0.0.1:8702-15) [e978fa0] ResourceManager::AddVds - VDS fca61124-29a6-11e2-a257-52540035ed48 was added to the Resource Manager >2012-11-08 08:19:55,481 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-15) [e978fa0] FINISH, AddVdsVDSCommand, log id: 5244279f >2012-11-08 08:19:55,501 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-4-thread-49) [a707348] Running command: InstallVdsCommand internal: true. Entities affected : ID: fca61124-29a6-11e2-a257-52540035ed48 Type: VDS >2012-11-08 08:19:55,502 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-4-thread-49) [a707348] Before Installation pool-4-thread-49 >2012-11-08 08:19:55,502 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-49) [a707348] Installation of 10.70.36.19. Executing installation stage. (Stage: Starting Host installation) >2012-11-08 08:19:55,502 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-49) [a707348] Installation of 10.70.36.19. Executing installation stage. (Stage: Connecting to Host) >2012-11-08 08:19:55,569 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-49) [a707348] Installation of 10.70.36.19. Received message: <BSTRAP component='RHEV_INSTALL' status='OK' message='Connected to Host 10.70.36.19 with SSH key fingerprint: 74:1f:5c:d9:6d:8e:d7:72:33:74:37:01:2c:d8:1a:44'/>. FYI. (Stage: Connecting to Host) >2012-11-08 08:19:55,602 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-49) [a707348] Installation of 10.70.36.19. Successfully connected to server ssh. (Stage: Connecting to Host) >2012-11-08 08:19:55,602 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-49) [a707348] Installation of 10.70.36.19. Executing installation stage. (Stage: Get the unique vds id) >2012-11-08 08:19:55,602 INFO [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-4-thread-49) SSH execute 10.70.36.19 'IDFILE=/etc/vdsm/vdsm.id; if [ -r "${IDFILE}" ]; then cat "${IDFILE}"; else UUID="$(dmidecode -s system-uuid 2> /dev/null | sed -e 's/.*Not.*//' )"; if [ -z "${UUID}" ]; then UUID="$(uuidgen 2> /dev/null)" && mkdir -p "$(dirname "${IDFILE}")" && echo "${UUID}" > "${IDFILE}" && chmod 0644 "${IDFILE}"; fi; [ -n "${UUID}" ] && echo "${UUID}"; fi' >2012-11-08 08:19:55,718 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-69) Installation of 10.70.36.19. Received message: 00000000-0000-0000-0000-0025902F4FF8. FYI. (Stage: Get the unique vds id) >2012-11-08 08:19:55,719 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-69) Installation of 10.70.36.19. Assigning unique id 00000000-0000-0000-0000-0025902F4FF8 to Host. (Stage: Get the unique vds id) >2012-11-08 08:19:55,722 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-49) [a707348] Installation of 10.70.36.19. Executing installation stage. (Stage: Upload Installation script to Host) >2012-11-08 08:19:55,722 INFO [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-4-thread-49) SSH execute 10.70.36.19 'cat > '/tmp/ovirt-id_rsa_d3a65d1e-e54a-4810-890e-6cbdef7fdd9a'' >2012-11-08 08:19:55,806 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-49) [a707348] Installation of 10.70.36.19. Executing installation stage. (Stage: Running first installation script on Host) >2012-11-08 08:19:55,806 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-49) [a707348] Installation of 10.70.36.19. Sending SSH Command umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar -C "${MYTMP}" --no-same-permissions -o -x && "${MYTMP}"/setup -c 'ssl=true;management_port=54321' -O 'lab.eng.blr.redhat.com' -t 2012-11-08T13:19:55 -S /tmp/ovirt-id_rsa_d3a65d1e-e54a-4810-890e-6cbdef7fdd9a -p 80 -b -B rhevm http://guido.lab.eng.blr.redhat.com:80/Components/vds/ http://guido.lab.eng.blr.redhat.com:80/Components/vds/ 10.70.36.19 d3a65d1e-e54a-4810-890e-6cbdef7fdd9a False < /var/cache/ovirt-engine/vdsm-bootstrap-2.tar. (Stage: Running first installation script on Host) >2012-11-08 08:19:55,807 INFO [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-4-thread-49) SSH execute 10.70.36.19 'umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar -C "${MYTMP}" --no-same-permissions -o -x && "${MYTMP}"/setup -c 'ssl=true;management_port=54321' -O 'lab.eng.blr.redhat.com' -t 2012-11-08T13:19:55 -S /tmp/ovirt-id_rsa_d3a65d1e-e54a-4810-890e-6cbdef7fdd9a -p 80 -b -B rhevm http://guido.lab.eng.blr.redhat.com:80/Components/vds/ http://guido.lab.eng.blr.redhat.com:80/Components/vds/ 10.70.36.19 d3a65d1e-e54a-4810-890e-6cbdef7fdd9a False' >2012-11-08 08:19:55,910 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-69) Installation of 10.70.36.19. Received message: <BSTRAP component='INSTALLER' status='OK' message='Test platform succeeded'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:19:55,941 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-68) DbUserCacheManager::refreshAllUserData() - entered >2012-11-08 08:19:56,607 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (ajp-/127.0.0.1:8702-8) [4efb64d4] FINISH, ConnectStoragePoolVDSCommand, log id: 31b53a6d >2012-11-08 08:19:56,635 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (ajp-/127.0.0.1:8702-8) [4efb64d4] hostFromVds::Connected host to pool - selectedVds - rhs-arch-srv2.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-FUNCTIONAL >2012-11-08 08:19:56,638 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (ajp-/127.0.0.1:8702-8) [4efb64d4] starting spm on vds rhs-arch-srv2.lab.eng.blr.redhat.com, storage pool RHS-FUNCTIONAL, prevId -1, LVER -1 >2012-11-08 08:19:56,639 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (ajp-/127.0.0.1:8702-8) [4efb64d4] START, SpmStartVDSCommand(HostName = rhs-arch-srv2.lab.eng.blr.redhat.com, HostId = cf017816-29a2-11e2-949e-52540035ed48, storagePoolId = f4c98a0b-0578-485f-90f4-743df3d05d7e, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4ee8282c >2012-11-08 08:19:56,669 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (ajp-/127.0.0.1:8702-8) [4efb64d4] spmStart polling started: taskId = eed6477d-fd81-4d45-8a40-a3bdebf7b822 >2012-11-08 08:19:57,018 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-69) Installation of 10.70.36.19. Received message: <BSTRAP component='SET_SYSTEM_TIME' status='OK' message='setSystemTime ended successfully'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:19:57,482 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-70) Initializing Host: rhs-gp-srv15.lab.eng.blr.redhat.com >2012-11-08 08:19:59,413 INFO [org.ovirt.engine.core.bll.LogoutUserCommand] (ajp-/127.0.0.1:8702-12) [4a71999a] Running command: LogoutUserCommand internal: false. >2012-11-08 08:20:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-75) Autorecovering hosts is disabled, skipping >2012-11-08 08:20:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-75) Autorecovering storage domains is disabled, skipping >2012-11-08 08:20:00,444 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-69) Installation of 10.70.36.19. Received message: <BSTRAP component='RHN_REGISTRATION' status='OK' message='Host properly registered with RHN/Satellite.'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:20:00,613 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-69) Installation of 10.70.36.19. Received message: <BSTRAP component='VDSM_MAJOR_VER' status='OK' message='Available VDSM matches requirements'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:20:00,618 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-69) Installation of 10.70.36.19. Received message: <BSTRAP component='VT_SVM' status='OK' processor='Intel' message='Server supports virtualization'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:20:00,622 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-69) Installation of 10.70.36.19. Received message: <BSTRAP component='OS' status='OK' type='RHEL6' message='Supported platform version'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:20:00,626 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-69) Installation of 10.70.36.19. Received message: <BSTRAP component='KERNEL' status='OK' version='279' message='Supported kernel version: 279'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:20:00,685 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-69) Installation of 10.70.36.19. Received message: <BSTRAP component='REQ PACKAGES' status='WARN' result='SDL.x86_64' message='package SDL.x86_64 is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:20:00,699 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-69) Installation of 10.70.36.19. Received message: <BSTRAP component='REQ PACKAGES' status='WARN' result='bridge-utils.x86_64' message='package bridge-utils.x86_64 is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:20:00,712 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-69) Installation of 10.70.36.19. Received message: <BSTRAP component='REQ PACKAGES' status='WARN' result='mesa-libGLU.x86_64' message='package mesa-libGLU.x86_64 is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:20:00,727 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-69) Installation of 10.70.36.19. Received message: <BSTRAP component='REQ PACKAGES' status='OK' result='openssl.x86_64' message='openssl-1.0.0-25.el6_3.1.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:20:00,742 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-69) Installation of 10.70.36.19. Received message: <BSTRAP component='REQ PACKAGES' status='OK' result='m2crypto.x86_64' message='m2crypto-0.20.2-9.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:20:00,755 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-69) Installation of 10.70.36.19. Received message: <BSTRAP component='REQ PACKAGES' status='WARN' result='rsync.x86_64' message='package rsync.x86_64 is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:20:00,769 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-69) Installation of 10.70.36.19. Received message: <BSTRAP component='VDS PACKAGES' status='WARN' result='qemu-kvm' message='package qemu-kvm is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:20:00,782 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-69) Installation of 10.70.36.19. Received message: <BSTRAP component='VDS PACKAGES' status='WARN' result='qemu-kvm-tools' message='package qemu-kvm-tools is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:20:00,795 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-69) Installation of 10.70.36.19. Received message: <BSTRAP component='VDS PACKAGES' status='WARN' result='(~vdsm~, ~vdsm-cli~)' message='package vdsm is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:20:00,808 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-69) Installation of 10.70.36.19. Received message: <BSTRAP component='VDS PACKAGES' status='WARN' result='libjpeg' message='package libjpeg is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:20:00,821 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-69) Installation of 10.70.36.19. Received message: <BSTRAP component='VDS PACKAGES' status='WARN' result='spice-server' message='package spice-server is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:20:00,834 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-69) Installation of 10.70.36.19. Received message: <BSTRAP component='VDS PACKAGES' status='WARN' result='pixman' message='package pixman is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:20:00,847 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-69) Installation of 10.70.36.19. Received message: <BSTRAP component='VDS PACKAGES' status='WARN' result='seabios' message='package seabios is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:20:00,860 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-69) Installation of 10.70.36.19. Received message: <BSTRAP component='VDS PACKAGES' status='WARN' result='qemu-img' message='package qemu-img is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:20:00,873 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-69) Installation of 10.70.36.19. Received message: <BSTRAP component='VDS PACKAGES' status='WARN' result='fence-agents' message='package fence-agents is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:20:00,886 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-69) Installation of 10.70.36.19. Received message: <BSTRAP component='VDS PACKAGES' status='WARN' result='libselinux-python' message='package libselinux-python is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:20:00,899 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-69) Installation of 10.70.36.19. Received message: <BSTRAP component='VDS PACKAGES' status='WARN' result='sanlock' message='package sanlock is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:20:00,912 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-69) Installation of 10.70.36.19. Received message: <BSTRAP component='VDS PACKAGES' status='WARN' result='sanlock-python' message='package sanlock-python is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:20:06,644 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-95) VM disrep1 661d6af9-9024-4744-91b1-258e38466815 moved from PoweringUp --> Up >2012-11-08 08:20:06,673 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsDynamicDataVDSCommand] (QuartzScheduler_Worker-95) START, UpdateVdsDynamicDataVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vdsDynamic=org.ovirt.engine.core.common.businessentities.VdsDynamic@ffde4263), log id: 2c8891d0 >2012-11-08 08:20:06,674 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsDynamicDataVDSCommand] (QuartzScheduler_Worker-95) FINISH, UpdateVdsDynamicDataVDSCommand, log id: 2c8891d0 >2012-11-08 08:20:18,218 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (ajp-/127.0.0.1:8702-8) [4efb64d4] spmStart polling ended: taskId = eed6477d-fd81-4d45-8a40-a3bdebf7b822 task status = finished >2012-11-08 08:20:18,239 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (ajp-/127.0.0.1:8702-8) [4efb64d4] spmStart polling ended, spm status: SPM >2012-11-08 08:20:18,241 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (ajp-/127.0.0.1:8702-8) [4efb64d4] START, HSMClearTaskVDSCommand(HostName = rhs-arch-srv2.lab.eng.blr.redhat.com, HostId = cf017816-29a2-11e2-949e-52540035ed48, taskId=eed6477d-fd81-4d45-8a40-a3bdebf7b822), log id: 23fc0bf6 >2012-11-08 08:20:18,265 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (ajp-/127.0.0.1:8702-8) [4efb64d4] FINISH, HSMClearTaskVDSCommand, log id: 23fc0bf6 >2012-11-08 08:20:18,265 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (ajp-/127.0.0.1:8702-8) [4efb64d4] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@787555ee, log id: 4ee8282c >2012-11-08 08:20:18,267 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (ajp-/127.0.0.1:8702-8) [4efb64d4] Initialize Irs proxy from vds: 10.70.36.2 >2012-11-08 08:20:18,305 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-8) [4efb64d4] FINISH, ActivateStorageDomainVDSCommand, log id: be646ee >2012-11-08 08:20:18,305 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp-/127.0.0.1:8702-8) [4efb64d4] ActivateStorage Domain. After Connect all hosts to pool. Time:11/8/12 8:20 AM >2012-11-08 08:20:18,322 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-10) [4efb64d4] START, ConnectStoragePoolVDSCommand(HostName = rhs-arch-srv2.lab.eng.blr.redhat.com, HostId = cf017816-29a2-11e2-949e-52540035ed48, storagePoolId = f4c98a0b-0578-485f-90f4-743df3d05d7e, vds_spm_id = 2, masterDomainId = 00068b9a-c8e3-4e01-b2c4-963931bfae65, masterVersion = 1), log id: 3d1881df >2012-11-08 08:20:18,345 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-10) [4efb64d4] FINISH, ConnectStoragePoolVDSCommand, log id: 3d1881df >2012-11-08 08:20:18,360 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp-/127.0.0.1:8702-8) [4efb64d4] ActivateStorage Domain. After change storage pool status in vds. Time:11/8/12 8:20 AM >2012-11-08 08:20:33,421 INFO [org.ovirt.engine.core.bll.storage.AttachStorageDomainToPoolCommand] (pool-4-thread-10) [6c95ee00] Running command: AttachStorageDomainToPoolCommand internal: false. Entities affected : ID: 7a268ac7-857a-4c53-966d-247deb509161 Type: Storage >2012-11-08 08:20:33,441 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (pool-4-thread-50) [62b5d102] START, ValidateStorageServerConnectionVDSCommand(HostName = rhs-arch-srv2.lab.eng.blr.redhat.com, HostId = cf017816-29a2-11e2-949e-52540035ed48, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 658918ca-5dd4-4f39-9d39-f4a117794418, connection: guido.lab.eng.blr.redhat.com:/iso2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: e145020 >2012-11-08 08:20:33,460 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (pool-4-thread-50) [62b5d102] FINISH, ValidateStorageServerConnectionVDSCommand, return: {658918ca-5dd4-4f39-9d39-f4a117794418=0}, log id: e145020 >2012-11-08 08:20:33,460 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (pool-4-thread-50) [62b5d102] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2012-11-08 08:20:33,471 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-4-thread-50) [62b5d102] START, ConnectStorageServerVDSCommand(HostName = rhs-arch-srv2.lab.eng.blr.redhat.com, HostId = cf017816-29a2-11e2-949e-52540035ed48, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 658918ca-5dd4-4f39-9d39-f4a117794418, connection: guido.lab.eng.blr.redhat.com:/iso2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 5070a557 >2012-11-08 08:20:33,629 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-4-thread-50) [62b5d102] FINISH, ConnectStorageServerVDSCommand, return: {658918ca-5dd4-4f39-9d39-f4a117794418=0}, log id: 5070a557 >2012-11-08 08:20:33,630 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.AttachStorageDomainVDSCommand] (pool-4-thread-10) [6c95ee00] START, AttachStorageDomainVDSCommand( storagePoolId = f4c98a0b-0578-485f-90f4-743df3d05d7e, ignoreFailoverLimit = false, compatabilityVersion = null, storageDomainId = 7a268ac7-857a-4c53-966d-247deb509161), log id: 1c62e3ea >2012-11-08 08:20:39,203 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.AttachStorageDomainVDSCommand] (pool-4-thread-10) [6c95ee00] FINISH, AttachStorageDomainVDSCommand, log id: 1c62e3ea >2012-11-08 08:20:48,326 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-23) VM disrep2 8a0d2e26-440c-4d6c-91d3-1fc4d9b75990 moved from PoweringUp --> Up >2012-11-08 08:20:48,353 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsDynamicDataVDSCommand] (QuartzScheduler_Worker-23) START, UpdateVdsDynamicDataVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vdsDynamic=org.ovirt.engine.core.common.businessentities.VdsDynamic@ad5d057e), log id: 23168998 >2012-11-08 08:20:48,354 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsDynamicDataVDSCommand] (QuartzScheduler_Worker-23) FINISH, UpdateVdsDynamicDataVDSCommand, log id: 23168998 >2012-11-08 08:20:50,436 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-61) Installation of 10.70.36.16. Received message: <BSTRAP component='REQ PACKAGES' status='OK' result='rsync.x86_64' message='rsync-3.0.6-9.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:21:00,702 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-69) Installation of 10.70.36.19. Received message: <BSTRAP component='REQ PACKAGES' status='OK' result='rsync.x86_64' message='rsync-3.0.6-9.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:21:27,873 INFO [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (ajp-/127.0.0.1:8702-9) [694a73cc] Lock Acquired to object EngineLock [exclusiveLocks= key: FirstVM value: VM_NAME >, sharedLocks= ] >2012-11-08 08:21:27,889 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-9) [694a73cc] START, IsValidVDSCommand( storagePoolId = f4c98a0b-0578-485f-90f4-743df3d05d7e, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3fac2439 >2012-11-08 08:21:27,889 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-9) [694a73cc] FINISH, IsValidVDSCommand, return: true, log id: 3fac2439 >2012-11-08 08:21:27,893 INFO [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (ajp-/127.0.0.1:8702-9) [694a73cc] Running command: AddVmFromScratchCommand internal: false. Entities affected : ID: f9f22222-2976-11e2-a2cd-52540035ed48 Type: VdsGroups >2012-11-08 08:21:27,894 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-9) [694a73cc] START, IsValidVDSCommand( storagePoolId = f4c98a0b-0578-485f-90f4-743df3d05d7e, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 5eafad69 >2012-11-08 08:21:27,894 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-9) [694a73cc] FINISH, IsValidVDSCommand, return: true, log id: 5eafad69 >2012-11-08 08:21:27,917 INFO [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (ajp-/127.0.0.1:8702-9) [694a73cc] Lock freed to object EngineLock [exclusiveLocks= key: FirstVM value: VM_NAME >, sharedLocks= ] >2012-11-08 08:21:32,696 INFO [org.ovirt.engine.core.bll.AddVmInterfaceCommand] (ajp-/127.0.0.1:8702-11) [e81fe65] Running command: AddVmInterfaceCommand internal: false. Entities affected : ID: 3b031d10-a679-4211-88d6-6b31c6dd7537 Type: VM >2012-11-08 08:21:32,698 INFO [org.ovirt.engine.core.bll.MacPoolManager] (ajp-/127.0.0.1:8702-11) [e81fe65] MacPoolManager::allocateNewMac entered >2012-11-08 08:21:32,698 INFO [org.ovirt.engine.core.bll.MacPoolManager] (ajp-/127.0.0.1:8702-11) [e81fe65] MacPoolManager::allocateNewMac allocated mac = 00:1a:4a:46:22:61 >2012-11-08 08:21:32,731 INFO [org.ovirt.engine.core.bll.ActivateDeactivateVmNicCommand] (ajp-/127.0.0.1:8702-11) [30e978f0] Running command: ActivateDeactivateVmNicCommand internal: true. Entities affected : ID: 3b031d10-a679-4211-88d6-6b31c6dd7537 Type: VM >2012-11-08 08:21:36,030 INFO [org.ovirt.engine.core.bll.AddVmInterfaceCommand] (ajp-/127.0.0.1:8702-4) [2780e687] Running command: AddVmInterfaceCommand internal: false. Entities affected : ID: 3b031d10-a679-4211-88d6-6b31c6dd7537 Type: VM >2012-11-08 08:21:36,031 INFO [org.ovirt.engine.core.bll.MacPoolManager] (ajp-/127.0.0.1:8702-4) [2780e687] MacPoolManager::allocateNewMac entered >2012-11-08 08:21:36,031 INFO [org.ovirt.engine.core.bll.MacPoolManager] (ajp-/127.0.0.1:8702-4) [2780e687] MacPoolManager::allocateNewMac allocated mac = 00:1a:4a:46:22:62 >2012-11-08 08:21:36,056 INFO [org.ovirt.engine.core.bll.ActivateDeactivateVmNicCommand] (ajp-/127.0.0.1:8702-4) [29f39db2] Running command: ActivateDeactivateVmNicCommand internal: true. Entities affected : ID: 3b031d10-a679-4211-88d6-6b31c6dd7537 Type: VM >2012-11-08 08:21:44,983 INFO [org.ovirt.engine.core.bll.AddDiskCommand] (ajp-/127.0.0.1:8702-5) [3a7d8cff] Lock Acquired to object EngineLock [exclusiveLocks= key: 3b031d10-a679-4211-88d6-6b31c6dd7537 value: VM_DISK_BOOT >, sharedLocks= key: 3b031d10-a679-4211-88d6-6b31c6dd7537 value: VM >] >2012-11-08 08:21:44,986 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-5) [3a7d8cff] START, IsValidVDSCommand( storagePoolId = f4c98a0b-0578-485f-90f4-743df3d05d7e, ignoreFailoverLimit = false, compatabilityVersion = null), log id: c828727 >2012-11-08 08:21:44,987 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-5) [3a7d8cff] FINISH, IsValidVDSCommand, return: true, log id: c828727 >2012-11-08 08:21:44,993 INFO [org.ovirt.engine.core.bll.AddDiskCommand] (ajp-/127.0.0.1:8702-5) [3a7d8cff] Running command: AddDiskCommand internal: false. Entities affected : ID: 3b031d10-a679-4211-88d6-6b31c6dd7537 Type: VM, ID: 00068b9a-c8e3-4e01-b2c4-963931bfae65 Type: Storage >2012-11-08 08:21:45,013 INFO [org.ovirt.engine.core.bll.AddImageFromScratchCommand] (ajp-/127.0.0.1:8702-5) [6af4e138] Running command: AddImageFromScratchCommand internal: true. Entities affected : ID: 00068b9a-c8e3-4e01-b2c4-963931bfae65 Type: Storage >2012-11-08 08:21:45,028 INFO [org.ovirt.engine.core.bll.AddImageFromScratchCommand] (ajp-/127.0.0.1:8702-5) [6af4e138] Lock freed to object EngineLock [exclusiveLocks= key: 3b031d10-a679-4211-88d6-6b31c6dd7537 value: VM_DISK_BOOT >, sharedLocks= key: 3b031d10-a679-4211-88d6-6b31c6dd7537 value: VM >] >2012-11-08 08:21:45,029 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (ajp-/127.0.0.1:8702-5) [6af4e138] START, CreateImageVDSCommand( storagePoolId = f4c98a0b-0578-485f-90f4-743df3d05d7e, ignoreFailoverLimit = false, compatabilityVersion = 3.1, storageDomainId = 00068b9a-c8e3-4e01-b2c4-963931bfae65, imageGroupId = 65c401eb-e12d-4d68-9fb3-e8237170875a, imageSizeInBytes = 10737418240, volumeFormat = RAW, newImageId = 13692da5-c6b6-494f-a94f-b295eeee4833, newImageDescription = ), log id: 5ce2d93f >2012-11-08 08:21:45,029 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (ajp-/127.0.0.1:8702-5) [6af4e138] -- CreateImageVDSCommand::ExecuteIrsBrokerCommand: calling 'createVolume' with two new parameters: description and UUID >2012-11-08 08:21:45,029 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (ajp-/127.0.0.1:8702-5) [6af4e138] -- createVolume parameters: > sdUUID=00068b9a-c8e3-4e01-b2c4-963931bfae65 > spUUID=f4c98a0b-0578-485f-90f4-743df3d05d7e > imgGUID=65c401eb-e12d-4d68-9fb3-e8237170875a > size=10,737,418,240 bytes > volFormat=RAW > volType=Sparse > volUUID=13692da5-c6b6-494f-a94f-b295eeee4833 > descr= > srcImgGUID=00000000-0000-0000-0000-000000000000 > srcVolUUID=00000000-0000-0000-0000-000000000000 > >2012-11-08 08:21:45,189 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (ajp-/127.0.0.1:8702-5) [6af4e138] FINISH, CreateImageVDSCommand, return: 13692da5-c6b6-494f-a94f-b295eeee4833, log id: 5ce2d93f >2012-11-08 08:21:45,192 INFO [org.ovirt.engine.core.bll.EntityAsyncTask] (ajp-/127.0.0.1:8702-5) [6af4e138] EntityAsyncTask::Adding EntityMultiAsyncTasks object for entity 65c401eb-e12d-4d68-9fb3-e8237170875a >2012-11-08 08:21:45,192 INFO [org.ovirt.engine.core.bll.EntityMultiAsyncTasks] (ajp-/127.0.0.1:8702-5) [6af4e138] EntityMultiAsyncTasks::AttachTask: Attaching task 34da2d65-3c6b-42b2-8e38-4ef7d535c4a1 to entity 65c401eb-e12d-4d68-9fb3-e8237170875a. >2012-11-08 08:21:45,193 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskUtils] (ajp-/127.0.0.1:8702-5) [6af4e138] Adding task 34da2d65-3c6b-42b2-8e38-4ef7d535c4a1 to DataBase >2012-11-08 08:21:45,207 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (ajp-/127.0.0.1:8702-5) [6af4e138] Adding task 34da2d65-3c6b-42b2-8e38-4ef7d535c4a1 (Parent Command AddDisk, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters), polling hasn't started yet.. >2012-11-08 08:21:45,215 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (ajp-/127.0.0.1:8702-5) [6af4e138] BaseAsyncTask::StartPollingTask: Starting to poll task 34da2d65-3c6b-42b2-8e38-4ef7d535c4a1. >2012-11-08 08:21:45,215 INFO [org.ovirt.engine.core.bll.EntityMultiAsyncTasks] (ajp-/127.0.0.1:8702-5) [6af4e138] EntityMultiAsyncTasks::StartPollingTask: Current Action Type for entity 65c401eb-e12d-4d68-9fb3-e8237170875a is AddDisk (determined by task 34da2d65-3c6b-42b2-8e38-4ef7d535c4a1) >2012-11-08 08:21:48,735 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-93) [23ef92ac] Polling and updating Async Tasks: 1 tasks, 1 tasks to poll now >2012-11-08 08:21:48,762 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (QuartzScheduler_Worker-93) [23ef92ac] SPMAsyncTask::PollTask: Polling task 34da2d65-3c6b-42b2-8e38-4ef7d535c4a1 (Parent Command AddDisk, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status finished, result 'success'. >2012-11-08 08:21:48,766 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (QuartzScheduler_Worker-93) [23ef92ac] BaseAsyncTask::OnTaskEndSuccess: Task 34da2d65-3c6b-42b2-8e38-4ef7d535c4a1 (Parent Command AddDisk, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended successfully. >2012-11-08 08:21:48,767 INFO [org.ovirt.engine.core.bll.EntityAsyncTask] (QuartzScheduler_Worker-93) [23ef92ac] EntityAsyncTask::EndActionIfNecessary: All tasks of entity 65c401eb-e12d-4d68-9fb3-e8237170875a has ended -> executing EndAction >2012-11-08 08:21:48,767 INFO [org.ovirt.engine.core.bll.EntityAsyncTask] (QuartzScheduler_Worker-93) [23ef92ac] EntityAsyncTask::EndAction: Ending action for 1 tasks (entity ID: 65c401eb-e12d-4d68-9fb3-e8237170875a): calling EndAction for action type AddDisk. >2012-11-08 08:21:48,767 INFO [org.ovirt.engine.core.bll.EntityAsyncTask] (pool-4-thread-50) [23ef92ac] EntityAsyncTask::EndCommandAction [within thread]context: Attempting to EndAction AddDisk >2012-11-08 08:21:48,775 INFO [org.ovirt.engine.core.bll.AddDiskCommand] (pool-4-thread-50) [3a7d8cff] Ending command successfully: org.ovirt.engine.core.bll.AddDiskCommand >2012-11-08 08:21:48,775 INFO [org.ovirt.engine.core.bll.AddImageFromScratchCommand] (pool-4-thread-50) [6af4e138] Ending command successfully: org.ovirt.engine.core.bll.AddImageFromScratchCommand >2012-11-08 08:21:48,784 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (pool-4-thread-50) [6af4e138] START, GetImageInfoVDSCommand( storagePoolId = f4c98a0b-0578-485f-90f4-743df3d05d7e, ignoreFailoverLimit = false, compatabilityVersion = null, storageDomainId = 00068b9a-c8e3-4e01-b2c4-963931bfae65, imageGroupId = 65c401eb-e12d-4d68-9fb3-e8237170875a, imageId = 13692da5-c6b6-494f-a94f-b295eeee4833), log id: 2cb8b3db >2012-11-08 08:21:48,833 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (pool-4-thread-50) [6af4e138] FINISH, GetImageInfoVDSCommand, return: org.ovirt.engine.core.common.businessentities.DiskImage@717f9358, log id: 2cb8b3db >2012-11-08 08:21:48,860 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UpdateVMVDSCommand] (pool-4-thread-50) START, UpdateVMVDSCommand( storagePoolId = f4c98a0b-0578-485f-90f4-743df3d05d7e, ignoreFailoverLimit = false, compatabilityVersion = null, storageDomainId = 00000000-0000-0000-0000-000000000000, infoDictionary.size = 1), log id: 4a3a007c >2012-11-08 08:21:48,916 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UpdateVMVDSCommand] (pool-4-thread-50) FINISH, UpdateVMVDSCommand, log id: 4a3a007c >2012-11-08 08:21:48,928 INFO [org.ovirt.engine.core.bll.EntityAsyncTask] (pool-4-thread-50) EntityAsyncTask::HandleEndActionResult [within thread]: EndAction for action type AddDisk completed, handling the result. >2012-11-08 08:21:48,928 INFO [org.ovirt.engine.core.bll.EntityAsyncTask] (pool-4-thread-50) EntityAsyncTask::HandleEndActionResult [within thread]: EndAction for action type AddDisk succeeded, clearing tasks. >2012-11-08 08:21:48,931 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (pool-4-thread-50) SPMAsyncTask::ClearAsyncTask: Attempting to clear task 34da2d65-3c6b-42b2-8e38-4ef7d535c4a1 >2012-11-08 08:21:48,931 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (pool-4-thread-50) START, SPMClearTaskVDSCommand( storagePoolId = f4c98a0b-0578-485f-90f4-743df3d05d7e, ignoreFailoverLimit = false, compatabilityVersion = null, taskId = 34da2d65-3c6b-42b2-8e38-4ef7d535c4a1), log id: 3786bef4 >2012-11-08 08:21:48,932 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (pool-4-thread-50) START, HSMClearTaskVDSCommand(HostName = rhs-arch-srv2.lab.eng.blr.redhat.com, HostId = cf017816-29a2-11e2-949e-52540035ed48, taskId=34da2d65-3c6b-42b2-8e38-4ef7d535c4a1), log id: 510a5c5f >2012-11-08 08:21:48,974 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (pool-4-thread-50) FINISH, HSMClearTaskVDSCommand, log id: 510a5c5f >2012-11-08 08:21:48,974 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (pool-4-thread-50) FINISH, SPMClearTaskVDSCommand, log id: 3786bef4 >2012-11-08 08:21:48,976 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (pool-4-thread-50) BaseAsyncTask::RemoveTaskFromDB: Removed task 34da2d65-3c6b-42b2-8e38-4ef7d535c4a1 from DataBase >2012-11-08 08:21:48,976 INFO [org.ovirt.engine.core.bll.EntityAsyncTask] (pool-4-thread-50) EntityAsyncTask::HandleEndActionResult [within thread]: Removing EntityMultiAsyncTasks object for entity 65c401eb-e12d-4d68-9fb3-e8237170875a >2012-11-08 08:21:56,793 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-61) Installation of 10.70.36.16. Received message: <BSTRAP component='REQ PACKAGES' status='OK' result='mesa-libGLU.x86_64' message='mesa-libGLU-7.11-5.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:22:10,919 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-61) Installation of 10.70.36.16. Received message: <BSTRAP component='REQ PACKAGES' status='OK' result='bridge-utils.x86_64' message='bridge-utils-1.2-9.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:22:10,953 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-69) Installation of 10.70.36.19. Received message: <BSTRAP component='REQ PACKAGES' status='OK' result='mesa-libGLU.x86_64' message='mesa-libGLU-7.11-5.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:22:23,687 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-69) Installation of 10.70.36.19. Received message: <BSTRAP component='REQ PACKAGES' status='OK' result='bridge-utils.x86_64' message='bridge-utils-1.2-9.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:22:24,965 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-61) Installation of 10.70.36.16. Received message: <BSTRAP component='REQ PACKAGES' status='OK' result='SDL.x86_64' message='SDL-1.2.14-3.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:22:31,948 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-5) [523a5b4a] START, IsValidVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6e13d228 >2012-11-08 08:22:31,948 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-5) [523a5b4a] FINISH, IsValidVDSCommand, return: true, log id: 6e13d228 >2012-11-08 08:22:31,990 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp-/127.0.0.1:8702-5) [523a5b4a] START, IsVmDuringInitiatingVDSCommand( vmId = 0ed98fdf-e2d5-4bf6-b94a-3cba410c4c8f), log id: 5b9e3243 >2012-11-08 08:22:31,990 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp-/127.0.0.1:8702-5) [523a5b4a] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 5b9e3243 >2012-11-08 08:22:32,028 INFO [org.ovirt.engine.core.bll.RunVmOnceCommand] (ajp-/127.0.0.1:8702-5) [523a5b4a] Running command: RunVmOnceCommand internal: false. Entities affected : ID: 0ed98fdf-e2d5-4bf6-b94a-3cba410c4c8f Type: VM >2012-11-08 08:22:32,083 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsoPrefixVDSCommand] (ajp-/127.0.0.1:8702-5) [523a5b4a] START, IsoPrefixVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 7da3e0b2 >2012-11-08 08:22:32,083 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsoPrefixVDSCommand] (ajp-/127.0.0.1:8702-5) [523a5b4a] FINISH, IsoPrefixVDSCommand, return: /rhev/data-center/3a17dcac-a6e3-4563-a4b0-f27d89b68dd6/7a268ac7-857a-4c53-966d-247deb509161/images/11111111-1111-1111-1111-111111111111, log id: 7da3e0b2 >2012-11-08 08:22:32,102 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-5) [523a5b4a] START, CreateVmVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=0ed98fdf-e2d5-4bf6-b94a-3cba410c4c8f, vm=org.ovirt.engine.core.common.businessentities.VM@7fd9349e), log id: 11f5068f >2012-11-08 08:22:32,125 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8702-5) [523a5b4a] START, CreateVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=0ed98fdf-e2d5-4bf6-b94a-3cba410c4c8f, vm=org.ovirt.engine.core.common.businessentities.VM@7fd9349e), log id: 241eb852 >2012-11-08 08:22:32,176 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8702-5) [523a5b4a] org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand spiceSslCipherSuite=DEFAULT,memSize=512,kvmEnable=true,smp=1,emulatedMachine=rhel6.3.0,vmType=kvm,keyboardLayout=en-us,nice=0,display=qxl,smpCoresPerSocket=1,spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard,timeOffset=-43200,transparentHugePages=true,vmId=0ed98fdf-e2d5-4bf6-b94a-3cba410c4c8f,devices=[Ljava.util.Map;@5d34aabb,acpiEnable=true,vmName=disrep3,cpuType=Nehalem,custom={} >2012-11-08 08:22:32,176 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8702-5) [523a5b4a] FINISH, CreateVDSCommand, log id: 241eb852 >2012-11-08 08:22:32,176 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-5) [523a5b4a] IncreasePendingVms::CreateVmIncreasing vds rhs-gp-srv11.lab.eng.blr.redhat.com pending vcpu count, now 1. Vm: disrep3 >2012-11-08 08:22:32,179 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-5) [523a5b4a] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 11f5068f >2012-11-08 08:22:33,292 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-29) VM disrep3 0ed98fdf-e2d5-4bf6-b94a-3cba410c4c8f moved from WaitForLaunch --> PoweringUp >2012-11-08 08:22:33,292 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (QuartzScheduler_Worker-29) START, FullListVdsCommand( HostId = 96739be4-2990-11e2-8577-52540035ed48, vds=null, vmIds=[0ed98fdf-e2d5-4bf6-b94a-3cba410c4c8f]), log id: 64f446f6 >2012-11-08 08:22:33,320 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (QuartzScheduler_Worker-29) FINISH, FullListVdsCommand, return: [Lorg.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcStruct;@51078790, log id: 64f446f6 >2012-11-08 08:22:37,377 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-69) Installation of 10.70.36.19. Received message: <BSTRAP component='REQ PACKAGES' status='OK' result='SDL.x86_64' message='SDL-1.2.14-3.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:22:39,908 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-61) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='sanlock-python' message='sanlock-python-2.3-4.el6_3.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:22:50,345 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-61) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='sanlock' message='sanlock-2.3-4.el6_3.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:22:51,424 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-69) Installation of 10.70.36.19. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='sanlock-python' message='sanlock-python-2.3-4.el6_3.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:22:56,294 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-62) VM b2 68655de5-7bd9-4bdc-8628-3b2069ec312f moved from Up --> RebootInProgress >2012-11-08 08:22:56,295 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (QuartzScheduler_Worker-62) START, FullListVdsCommand( HostId = 96739be4-2990-11e2-8577-52540035ed48, vds=null, vmIds=[68655de5-7bd9-4bdc-8628-3b2069ec312f]), log id: 3fdbcbbf >2012-11-08 08:22:56,324 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (QuartzScheduler_Worker-62) FINISH, FullListVdsCommand, return: [Lorg.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcStruct;@38f14c88, log id: 3fdbcbbf >2012-11-08 08:22:56,373 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-96) Setting new tasks map. The map contains now 0 tasks >2012-11-08 08:22:56,373 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-96) Cleared all tasks of pool f4c98a0b-0578-485f-90f4-743df3d05d7e. >2012-11-08 08:22:56,375 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-96) Could not find vds that is spm and non-operational. >2012-11-08 08:23:02,105 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-69) Installation of 10.70.36.19. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='sanlock' message='sanlock-2.3-4.el6_3.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:23:03,888 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-61) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='libselinux-python' message='libselinux-python-2.0.94-5.3.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:23:13,977 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp-/127.0.0.1:8702-5) Lock Acquired to object EngineLock [exclusiveLocks= key: 7a268ac7-857a-4c53-966d-247deb509161 value: STORAGE >, sharedLocks= ] >2012-11-08 08:23:14,000 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (pool-4-thread-10) [687b0af0] Running command: ActivateStorageDomainCommand internal: false. Entities affected : ID: 7a268ac7-857a-4c53-966d-247deb509161 Type: Storage >2012-11-08 08:23:14,005 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (pool-4-thread-10) [687b0af0] Lock freed to object EngineLock [exclusiveLocks= key: 7a268ac7-857a-4c53-966d-247deb509161 value: STORAGE >, sharedLocks= ] >2012-11-08 08:23:14,005 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (pool-4-thread-10) [687b0af0] ActivateStorage Domain. Before Connect all hosts to pool. Time:11/8/12 8:23 AM >2012-11-08 08:23:14,017 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (pool-4-thread-50) [1dd9ae37] START, ValidateStorageServerConnectionVDSCommand(HostName = rhs-arch-srv2.lab.eng.blr.redhat.com, HostId = cf017816-29a2-11e2-949e-52540035ed48, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 658918ca-5dd4-4f39-9d39-f4a117794418, connection: guido.lab.eng.blr.redhat.com:/iso2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 3dcb259 >2012-11-08 08:23:14,037 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (pool-4-thread-50) [1dd9ae37] FINISH, ValidateStorageServerConnectionVDSCommand, return: {658918ca-5dd4-4f39-9d39-f4a117794418=0}, log id: 3dcb259 >2012-11-08 08:23:14,037 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (pool-4-thread-50) [1dd9ae37] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2012-11-08 08:23:14,038 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-4-thread-50) [1dd9ae37] START, ConnectStorageServerVDSCommand(HostName = rhs-arch-srv2.lab.eng.blr.redhat.com, HostId = cf017816-29a2-11e2-949e-52540035ed48, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 658918ca-5dd4-4f39-9d39-f4a117794418, connection: guido.lab.eng.blr.redhat.com:/iso2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 3f0f5c62 >2012-11-08 08:23:14,074 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-4-thread-50) [1dd9ae37] FINISH, ConnectStorageServerVDSCommand, return: {658918ca-5dd4-4f39-9d39-f4a117794418=0}, log id: 3f0f5c62 >2012-11-08 08:23:14,075 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (pool-4-thread-10) [687b0af0] START, ActivateStorageDomainVDSCommand( storagePoolId = f4c98a0b-0578-485f-90f4-743df3d05d7e, ignoreFailoverLimit = false, compatabilityVersion = null, storageDomainId = 7a268ac7-857a-4c53-966d-247deb509161), log id: 5715c3fd >2012-11-08 08:23:14,130 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (pool-4-thread-10) [687b0af0] FINISH, ActivateStorageDomainVDSCommand, log id: 5715c3fd >2012-11-08 08:23:14,131 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (pool-4-thread-10) [687b0af0] ActivateStorage Domain. After Connect all hosts to pool. Time:11/8/12 8:23 AM >2012-11-08 08:23:14,150 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.RefreshStoragePoolVDSCommand] (pool-4-thread-45) [687b0af0] START, RefreshStoragePoolVDSCommand(HostName = rhs-arch-srv2.lab.eng.blr.redhat.com, HostId = cf017816-29a2-11e2-949e-52540035ed48, storagePoolId = f4c98a0b-0578-485f-90f4-743df3d05d7e, masterStorageDomainId=00068b9a-c8e3-4e01-b2c4-963931bfae65, msterVersion=1), log id: 59f11e31 >2012-11-08 08:23:15,802 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-69) Installation of 10.70.36.19. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='libselinux-python' message='libselinux-python-2.0.94-5.3.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:23:16,700 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.RefreshStoragePoolVDSCommand] (pool-4-thread-45) [687b0af0] FINISH, RefreshStoragePoolVDSCommand, log id: 59f11e31 >2012-11-08 08:23:16,700 INFO [org.ovirt.engine.core.bll.storage.RefreshPoolSingleAsyncOperation] (pool-4-thread-45) [687b0af0] Refreshed vds rhs-arch-srv2.lab.eng.blr.redhat.com in pool RHS-FUNCTIONAL >2012-11-08 08:23:16,702 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (pool-4-thread-10) [687b0af0] ActivateStorage Domain. After change storage pool status in vds. Time:11/8/12 8:23 AM >2012-11-08 08:23:16,703 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-45) [687b0af0] START, GetIsoListVDSCommand( storagePoolId = f4c98a0b-0578-485f-90f4-743df3d05d7e, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6d757777 >2012-11-08 08:23:16,705 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-45) [687b0af0] START, HsmGetIsoListVDSCommand(HostName = rhs-arch-srv2.lab.eng.blr.redhat.com, HostId = cf017816-29a2-11e2-949e-52540035ed48, storagePoolId=f4c98a0b-0578-485f-90f4-743df3d05d7e), log id: 744239b2 >2012-11-08 08:23:16,954 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-45) [687b0af0] FINISH, HsmGetIsoListVDSCommand, return: [RHEL6.3-20120613.2-Server-x86_64-DVD1.iso], log id: 744239b2 >2012-11-08 08:23:16,954 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-45) [687b0af0] FINISH, GetIsoListVDSCommand, return: [RHEL6.3-20120613.2-Server-x86_64-DVD1.iso], log id: 6d757777 >2012-11-08 08:23:16,958 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetFloppyListVDSCommand] (pool-4-thread-45) [687b0af0] START, GetFloppyListVDSCommand( storagePoolId = f4c98a0b-0578-485f-90f4-743df3d05d7e, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 77c1b1b7 >2012-11-08 08:23:16,986 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetFloppyListVDSCommand] (pool-4-thread-45) [687b0af0] FINISH, GetFloppyListVDSCommand, return: [], log id: 77c1b1b7 >2012-11-08 08:23:34,199 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-17) [36467d8f] VM disrep3 0ed98fdf-e2d5-4bf6-b94a-3cba410c4c8f moved from PoweringUp --> Up >2012-11-08 08:23:34,225 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsDynamicDataVDSCommand] (QuartzScheduler_Worker-17) [36467d8f] START, UpdateVdsDynamicDataVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vdsDynamic=org.ovirt.engine.core.common.businessentities.VdsDynamic@93804418), log id: 5ed5bf36 >2012-11-08 08:23:34,227 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsDynamicDataVDSCommand] (QuartzScheduler_Worker-17) [36467d8f] FINISH, UpdateVdsDynamicDataVDSCommand, log id: 5ed5bf36 >2012-11-08 08:23:35,424 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsoPrefixVDSCommand] (ajp-/127.0.0.1:8702-5) [72143d20] START, IsoPrefixVDSCommand( storagePoolId = f4c98a0b-0578-485f-90f4-743df3d05d7e, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 22c9635b >2012-11-08 08:23:35,461 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsoPrefixVDSCommand] (ajp-/127.0.0.1:8702-5) [72143d20] FINISH, IsoPrefixVDSCommand, return: /rhev/data-center/f4c98a0b-0578-485f-90f4-743df3d05d7e/7a268ac7-857a-4c53-966d-247deb509161/images/11111111-1111-1111-1111-111111111111, log id: 22c9635b >2012-11-08 08:23:35,487 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-5) [72143d20] START, IsValidVDSCommand( storagePoolId = f4c98a0b-0578-485f-90f4-743df3d05d7e, ignoreFailoverLimit = false, compatabilityVersion = null), log id: da1d6c7 >2012-11-08 08:23:35,488 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-5) [72143d20] FINISH, IsValidVDSCommand, return: true, log id: da1d6c7 >2012-11-08 08:23:35,498 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp-/127.0.0.1:8702-5) [72143d20] START, IsVmDuringInitiatingVDSCommand( vmId = 3b031d10-a679-4211-88d6-6b31c6dd7537), log id: 2daceeb >2012-11-08 08:23:35,498 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp-/127.0.0.1:8702-5) [72143d20] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 2daceeb >2012-11-08 08:23:35,531 INFO [org.ovirt.engine.core.bll.RunVmOnceCommand] (ajp-/127.0.0.1:8702-5) [72143d20] Running command: RunVmOnceCommand internal: false. Entities affected : ID: 3b031d10-a679-4211-88d6-6b31c6dd7537 Type: VM >2012-11-08 08:23:35,576 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-5) [72143d20] START, CreateVmVDSCommand(HostName = rhs-arch-srv2.lab.eng.blr.redhat.com, HostId = cf017816-29a2-11e2-949e-52540035ed48, vmId=3b031d10-a679-4211-88d6-6b31c6dd7537, vm=org.ovirt.engine.core.common.businessentities.VM@716e91b3), log id: 38f2fbf7 >2012-11-08 08:23:35,580 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8702-5) [72143d20] START, CreateVDSCommand(HostName = rhs-arch-srv2.lab.eng.blr.redhat.com, HostId = cf017816-29a2-11e2-949e-52540035ed48, vmId=3b031d10-a679-4211-88d6-6b31c6dd7537, vm=org.ovirt.engine.core.common.businessentities.VM@716e91b3), log id: 740b2449 >2012-11-08 08:23:35,663 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8702-5) [72143d20] org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand spiceSslCipherSuite=DEFAULT,memSize=4200,kvmEnable=true,smp=2,emulatedMachine=rhel6.3.0,vmType=kvm,keyboardLayout=en-us,pitReinjection=false,nice=0,display=qxl,smpCoresPerSocket=1,spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard,timeOffset=-43200,transparentHugePages=true,vmId=3b031d10-a679-4211-88d6-6b31c6dd7537,devices=[Ljava.util.Map;@37c9f71b,acpiEnable=true,vmName=FirstVM,cpuType=Nehalem,custom={} >2012-11-08 08:23:35,663 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8702-5) [72143d20] FINISH, CreateVDSCommand, log id: 740b2449 >2012-11-08 08:23:35,663 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-5) [72143d20] IncreasePendingVms::CreateVmIncreasing vds rhs-arch-srv2.lab.eng.blr.redhat.com pending vcpu count, now 2. Vm: FirstVM >2012-11-08 08:23:35,665 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-5) [72143d20] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 38f2fbf7 >2012-11-08 08:23:36,388 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (QuartzScheduler_Worker-18) [7ab5f38b] START, FullListVdsCommand( HostId = cf017816-29a2-11e2-949e-52540035ed48, vds=null, vmIds=[3b031d10-a679-4211-88d6-6b31c6dd7537]), log id: 2d17092f >2012-11-08 08:23:36,410 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (QuartzScheduler_Worker-18) [7ab5f38b] FINISH, FullListVdsCommand, return: [Lorg.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcStruct;@435c03c9, log id: 2d17092f >2012-11-08 08:23:36,414 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-18) [7ab5f38b] Recieved a qxl Device without an address when processing VM 3b031d10-a679-4211-88d6-6b31c6dd7537 devices, skipping device: {specParams={vram=65536}, device=qxl, type=video, deviceId=43d91c0c-f458-48b9-bff8-cdca0ea5f732} >2012-11-08 08:23:36,414 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-18) [7ab5f38b] Recieved a bridge Device without an address when processing VM 3b031d10-a679-4211-88d6-6b31c6dd7537 devices, skipping device: {bootOrder=3, nicModel=pv, specParams={}, device=bridge, macAddr=00:1a:4a:46:22:61, type=interface, deviceId=75a6935b-35f5-4424-a421-45e25a9d5b5b, filter=vdsm-no-mac-spoofing, network=rhevm} >2012-11-08 08:23:36,414 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-18) [7ab5f38b] Recieved a bridge Device without an address when processing VM 3b031d10-a679-4211-88d6-6b31c6dd7537 devices, skipping device: {bootOrder=4, nicModel=pv, specParams={}, device=bridge, macAddr=00:1a:4a:46:22:62, type=interface, deviceId=36ff1ef5-6141-440b-ba7a-28e057a1dcc4, filter=vdsm-no-mac-spoofing, network=rhevm} >2012-11-08 08:23:36,414 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-18) [7ab5f38b] Recieved a memballoon Device without an address when processing VM 3b031d10-a679-4211-88d6-6b31c6dd7537 devices, skipping device: {specParams={model=virtio}, device=memballoon, type=balloon, deviceId=aae49b7a-387e-4391-94bc-aa86438faf5b} >2012-11-08 08:23:36,414 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-18) [7ab5f38b] Recieved a cdrom Device without an address when processing VM 3b031d10-a679-4211-88d6-6b31c6dd7537 devices, skipping device: {shared=false, bootOrder=2, iface=ide, index=2, specParams={}, device=cdrom, path=/rhev/data-center/f4c98a0b-0578-485f-90f4-743df3d05d7e/7a268ac7-857a-4c53-966d-247deb509161/images/11111111-1111-1111-1111-111111111111/RHEL6.3-20120613.2-Server-x86_64-DVD1.iso, type=disk, readonly=true, deviceId=34864609-1d46-44a7-8c45-81751c135196} >2012-11-08 08:23:36,414 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-18) [7ab5f38b] Recieved a disk Device without an address when processing VM 3b031d10-a679-4211-88d6-6b31c6dd7537 devices, skipping device: {shared=false, index=0, volumeID=13692da5-c6b6-494f-a94f-b295eeee4833, propagateErrors=off, format=raw, type=disk, truesize=0, reqsize=0, iface=virtio, bootOrder=1, volumeChain=[Ljava.lang.Object;@33480335, imageID=65c401eb-e12d-4d68-9fb3-e8237170875a, domainID=00068b9a-c8e3-4e01-b2c4-963931bfae65, specParams={}, optional=false, path=/rhev/data-center/f4c98a0b-0578-485f-90f4-743df3d05d7e/00068b9a-c8e3-4e01-b2c4-963931bfae65/images/65c401eb-e12d-4d68-9fb3-e8237170875a/13692da5-c6b6-494f-a94f-b295eeee4833, device=disk, poolID=f4c98a0b-0578-485f-90f4-743df3d05d7e, readonly=false, deviceId=65c401eb-e12d-4d68-9fb3-e8237170875a, apparentsize=10737418240} >2012-11-08 08:23:38,487 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (QuartzScheduler_Worker-15) START, DestroyVDSCommand(HostName = rhs-arch-srv2.lab.eng.blr.redhat.com, HostId = cf017816-29a2-11e2-949e-52540035ed48, vmId=3b031d10-a679-4211-88d6-6b31c6dd7537, force=false, secondsToWait=0, gracefully=false), log id: 76631415 >2012-11-08 08:23:38,864 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (QuartzScheduler_Worker-15) FINISH, DestroyVDSCommand, log id: 76631415 >2012-11-08 08:23:38,893 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-15) Running on vds during rerun failed vm: null >2012-11-08 08:23:38,894 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-15) vm FirstVM running in db and not running in vds - add to rerun treatment. vds rhs-arch-srv2.lab.eng.blr.redhat.com >2012-11-08 08:23:38,899 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-15) Rerun vm 3b031d10-a679-4211-88d6-6b31c6dd7537. Called from vds rhs-arch-srv2.lab.eng.blr.redhat.com >2012-11-08 08:23:38,901 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsDynamicDataVDSCommand] (pool-4-thread-45) START, UpdateVdsDynamicDataVDSCommand(HostName = rhs-arch-srv2.lab.eng.blr.redhat.com, HostId = cf017816-29a2-11e2-949e-52540035ed48, vdsDynamic=org.ovirt.engine.core.common.businessentities.VdsDynamic@56b874d8), log id: 76091851 >2012-11-08 08:23:38,903 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsDynamicDataVDSCommand] (pool-4-thread-45) FINISH, UpdateVdsDynamicDataVDSCommand, log id: 76091851 >2012-11-08 08:23:38,930 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (pool-4-thread-45) START, IsValidVDSCommand( storagePoolId = f4c98a0b-0578-485f-90f4-743df3d05d7e, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 1906c3a0 >2012-11-08 08:23:38,931 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (pool-4-thread-45) FINISH, IsValidVDSCommand, return: true, log id: 1906c3a0 >2012-11-08 08:23:38,947 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (pool-4-thread-45) START, IsVmDuringInitiatingVDSCommand( vmId = 3b031d10-a679-4211-88d6-6b31c6dd7537), log id: 3af618b4 >2012-11-08 08:23:38,947 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (pool-4-thread-45) FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 3af618b4 >2012-11-08 08:23:38,979 INFO [org.ovirt.engine.core.bll.VdsSelector] (pool-4-thread-45) VDS rhs-gp-srv11.lab.eng.blr.redhat.com 96739be4-2990-11e2-8577-52540035ed48 is not in up status or belongs to the VM's cluster VDS rhs-gp-srv4.lab.eng.blr.redhat.com d2bd32d4-2992-11e2-ab59-52540035ed48 is not in up status or belongs to the VM's cluster VDS rhs-gp-srv15.lab.eng.blr.redhat.com fca61124-29a6-11e2-a257-52540035ed48 is not in up status or belongs to the VM's cluster VDS rhs-gp-srv12.lab.eng.blr.redhat.com f047378c-29a6-11e2-8a79-52540035ed48 is not in up status or belongs to the VM's cluster VDS rhs-arch-srv2.lab.eng.blr.redhat.com cf017816-29a2-11e2-949e-52540035ed48 is the same host the VM is currently running on >2012-11-08 08:23:38,979 WARN [org.ovirt.engine.core.bll.RunVmOnceCommand] (pool-4-thread-45) CanDoAction of action RunVmOnce failed. Reasons:VAR__ACTION__RUN,VAR__TYPE__VM,$VmName FirstVM,VAR__ACTION__RUN,VAR__TYPE__VM,ACTION_TYPE_FAILED_VDS_VM_CLUSTER >2012-11-08 08:23:46,146 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-15) Checking if user admin@internal is an admin, result true >2012-11-08 08:23:46,146 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-15) Running command: LoginUserCommand internal: false. >2012-11-08 08:23:46,162 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-15) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2012-11-08 08:23:48,844 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-42) VM b2 68655de5-7bd9-4bdc-8628-3b2069ec312f moved from RebootInProgress --> Up >2012-11-08 08:23:56,516 INFO [org.ovirt.engine.core.bll.LoginAdminUserCommand] (ajp-/127.0.0.1:8702-5) Checking if user admin@internal is an admin, result true >2012-11-08 08:23:56,516 INFO [org.ovirt.engine.core.bll.LoginAdminUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginAdminUserCommand internal: false. >2012-11-08 08:24:09,460 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-4) VM b1 67bd2d56-8e0b-4328-bcfd-ea7f4758c885 moved from Up --> RebootInProgress >2012-11-08 08:24:09,477 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (QuartzScheduler_Worker-4) START, FullListVdsCommand( HostId = 96739be4-2990-11e2-8577-52540035ed48, vds=null, vmIds=[67bd2d56-8e0b-4328-bcfd-ea7f4758c885]), log id: 1436a19 >2012-11-08 08:24:09,521 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (QuartzScheduler_Worker-4) FINISH, FullListVdsCommand, return: [Lorg.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcStruct;@1e479a1c, log id: 1436a19 >2012-11-08 08:24:19,935 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-44) VM b3 7fa6f92f-755f-44c3-8f35-b4c56410c87b moved from Up --> RebootInProgress >2012-11-08 08:24:19,939 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (QuartzScheduler_Worker-44) START, FullListVdsCommand( HostId = 96739be4-2990-11e2-8577-52540035ed48, vds=null, vmIds=[7fa6f92f-755f-44c3-8f35-b4c56410c87b]), log id: 1d3f42a1 >2012-11-08 08:24:19,959 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (QuartzScheduler_Worker-44) FINISH, FullListVdsCommand, return: [Lorg.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcStruct;@242f6707, log id: 1d3f42a1 >2012-11-08 08:24:31,142 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsoPrefixVDSCommand] (ajp-/127.0.0.1:8702-14) [2934da69] START, IsoPrefixVDSCommand( storagePoolId = f4c98a0b-0578-485f-90f4-743df3d05d7e, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 51bbcae1 >2012-11-08 08:24:31,143 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsoPrefixVDSCommand] (ajp-/127.0.0.1:8702-14) [2934da69] FINISH, IsoPrefixVDSCommand, return: /rhev/data-center/f4c98a0b-0578-485f-90f4-743df3d05d7e/7a268ac7-857a-4c53-966d-247deb509161/images/11111111-1111-1111-1111-111111111111, log id: 51bbcae1 >2012-11-08 08:24:31,201 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-14) [2934da69] START, IsValidVDSCommand( storagePoolId = f4c98a0b-0578-485f-90f4-743df3d05d7e, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 513e0332 >2012-11-08 08:24:31,201 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-14) [2934da69] FINISH, IsValidVDSCommand, return: true, log id: 513e0332 >2012-11-08 08:24:31,213 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp-/127.0.0.1:8702-14) [2934da69] START, IsVmDuringInitiatingVDSCommand( vmId = 3b031d10-a679-4211-88d6-6b31c6dd7537), log id: 511efe3f >2012-11-08 08:24:31,213 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp-/127.0.0.1:8702-14) [2934da69] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 511efe3f >2012-11-08 08:24:31,256 INFO [org.ovirt.engine.core.bll.RunVmOnceCommand] (ajp-/127.0.0.1:8702-14) [2934da69] Running command: RunVmOnceCommand internal: false. Entities affected : ID: 3b031d10-a679-4211-88d6-6b31c6dd7537 Type: VM >2012-11-08 08:24:31,297 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-14) [2934da69] START, CreateVmVDSCommand(HostName = rhs-arch-srv2.lab.eng.blr.redhat.com, HostId = cf017816-29a2-11e2-949e-52540035ed48, vmId=3b031d10-a679-4211-88d6-6b31c6dd7537, vm=org.ovirt.engine.core.common.businessentities.VM@85f628), log id: 6a3bdbb6 >2012-11-08 08:24:31,301 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8702-14) [2934da69] START, CreateVDSCommand(HostName = rhs-arch-srv2.lab.eng.blr.redhat.com, HostId = cf017816-29a2-11e2-949e-52540035ed48, vmId=3b031d10-a679-4211-88d6-6b31c6dd7537, vm=org.ovirt.engine.core.common.businessentities.VM@85f628), log id: a40e089 >2012-11-08 08:24:31,360 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8702-14) [2934da69] org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand spiceSslCipherSuite=DEFAULT,memSize=4200,kvmEnable=true,smp=2,emulatedMachine=rhel6.3.0,vmType=kvm,keyboardLayout=en-us,pitReinjection=false,nice=0,display=qxl,smpCoresPerSocket=1,spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard,timeOffset=-43200,transparentHugePages=true,vmId=3b031d10-a679-4211-88d6-6b31c6dd7537,devices=[Ljava.util.Map;@1a846741,acpiEnable=true,vmName=FirstVM,cpuType=Nehalem,custom={} >2012-11-08 08:24:31,360 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8702-14) [2934da69] FINISH, CreateVDSCommand, log id: a40e089 >2012-11-08 08:24:31,361 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-14) [2934da69] IncreasePendingVms::CreateVmIncreasing vds rhs-arch-srv2.lab.eng.blr.redhat.com pending vcpu count, now 2. Vm: FirstVM >2012-11-08 08:24:31,363 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-14) [2934da69] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 6a3bdbb6 >2012-11-08 08:24:33,794 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (QuartzScheduler_Worker-81) START, DestroyVDSCommand(HostName = rhs-arch-srv2.lab.eng.blr.redhat.com, HostId = cf017816-29a2-11e2-949e-52540035ed48, vmId=3b031d10-a679-4211-88d6-6b31c6dd7537, force=false, secondsToWait=0, gracefully=false), log id: 73cc9635 >2012-11-08 08:24:34,146 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (QuartzScheduler_Worker-81) FINISH, DestroyVDSCommand, log id: 73cc9635 >2012-11-08 08:24:34,168 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-81) Running on vds during rerun failed vm: null >2012-11-08 08:24:34,169 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-81) vm FirstVM running in db and not running in vds - add to rerun treatment. vds rhs-arch-srv2.lab.eng.blr.redhat.com >2012-11-08 08:24:34,172 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-81) Rerun vm 3b031d10-a679-4211-88d6-6b31c6dd7537. Called from vds rhs-arch-srv2.lab.eng.blr.redhat.com >2012-11-08 08:24:34,175 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsDynamicDataVDSCommand] (pool-4-thread-45) START, UpdateVdsDynamicDataVDSCommand(HostName = rhs-arch-srv2.lab.eng.blr.redhat.com, HostId = cf017816-29a2-11e2-949e-52540035ed48, vdsDynamic=org.ovirt.engine.core.common.businessentities.VdsDynamic@56b874d8), log id: 595573db >2012-11-08 08:24:34,177 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsDynamicDataVDSCommand] (pool-4-thread-45) FINISH, UpdateVdsDynamicDataVDSCommand, log id: 595573db >2012-11-08 08:24:34,200 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (pool-4-thread-45) START, IsValidVDSCommand( storagePoolId = f4c98a0b-0578-485f-90f4-743df3d05d7e, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 5dd6953 >2012-11-08 08:24:34,201 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (pool-4-thread-45) FINISH, IsValidVDSCommand, return: true, log id: 5dd6953 >2012-11-08 08:24:34,216 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (pool-4-thread-45) START, IsVmDuringInitiatingVDSCommand( vmId = 3b031d10-a679-4211-88d6-6b31c6dd7537), log id: 597f9d52 >2012-11-08 08:24:34,216 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (pool-4-thread-45) FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 597f9d52 >2012-11-08 08:24:34,252 INFO [org.ovirt.engine.core.bll.VdsSelector] (pool-4-thread-45) VDS rhs-gp-srv11.lab.eng.blr.redhat.com 96739be4-2990-11e2-8577-52540035ed48 is not in up status or belongs to the VM's cluster VDS rhs-gp-srv4.lab.eng.blr.redhat.com d2bd32d4-2992-11e2-ab59-52540035ed48 is not in up status or belongs to the VM's cluster VDS rhs-gp-srv15.lab.eng.blr.redhat.com fca61124-29a6-11e2-a257-52540035ed48 is not in up status or belongs to the VM's cluster VDS rhs-gp-srv12.lab.eng.blr.redhat.com f047378c-29a6-11e2-8a79-52540035ed48 is not in up status or belongs to the VM's cluster VDS rhs-arch-srv2.lab.eng.blr.redhat.com cf017816-29a2-11e2-949e-52540035ed48 is the same host the VM is currently running on >2012-11-08 08:24:34,252 WARN [org.ovirt.engine.core.bll.RunVmOnceCommand] (pool-4-thread-45) CanDoAction of action RunVmOnce failed. Reasons:VAR__ACTION__RUN,VAR__TYPE__VM,$VmName FirstVM,VAR__ACTION__RUN,VAR__TYPE__VM,ACTION_TYPE_FAILED_VDS_VM_CLUSTER >2012-11-08 08:24:54,017 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-12) [4ec16bb8] START, IsValidVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 1e263aa5 >2012-11-08 08:24:54,017 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-12) [4ec16bb8] FINISH, IsValidVDSCommand, return: true, log id: 1e263aa5 >2012-11-08 08:24:54,040 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp-/127.0.0.1:8702-12) [4ec16bb8] START, IsVmDuringInitiatingVDSCommand( vmId = e228f541-76b7-4a9b-b313-a6bacc2b73b9), log id: 7bf4543e >2012-11-08 08:24:54,040 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp-/127.0.0.1:8702-12) [4ec16bb8] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 7bf4543e >2012-11-08 08:24:54,074 INFO [org.ovirt.engine.core.bll.RunVmOnceCommand] (ajp-/127.0.0.1:8702-12) [4ec16bb8] Running command: RunVmOnceCommand internal: false. Entities affected : ID: e228f541-76b7-4a9b-b313-a6bacc2b73b9 Type: VM >2012-11-08 08:24:54,075 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-2) [663cb536] START, IsValidVDSCommand( storagePoolId = f4c98a0b-0578-485f-90f4-743df3d05d7e, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 56ad6d20 >2012-11-08 08:24:54,076 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-2) [663cb536] FINISH, IsValidVDSCommand, return: true, log id: 56ad6d20 >2012-11-08 08:24:54,079 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (ajp-/127.0.0.1:8702-2) [663cb536] START, GetImageInfoVDSCommand( storagePoolId = f4c98a0b-0578-485f-90f4-743df3d05d7e, ignoreFailoverLimit = false, compatabilityVersion = null, storageDomainId = 00068b9a-c8e3-4e01-b2c4-963931bfae65, imageGroupId = 65c401eb-e12d-4d68-9fb3-e8237170875a, imageId = 13692da5-c6b6-494f-a94f-b295eeee4833), log id: 20d9fc65 >2012-11-08 08:24:54,120 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsoPrefixVDSCommand] (ajp-/127.0.0.1:8702-12) [4ec16bb8] START, IsoPrefixVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 29c65025 >2012-11-08 08:24:54,120 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsoPrefixVDSCommand] (ajp-/127.0.0.1:8702-12) [4ec16bb8] FINISH, IsoPrefixVDSCommand, return: /rhev/data-center/3a17dcac-a6e3-4563-a4b0-f27d89b68dd6/7a268ac7-857a-4c53-966d-247deb509161/images/11111111-1111-1111-1111-111111111111, log id: 29c65025 >2012-11-08 08:24:54,129 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (ajp-/127.0.0.1:8702-2) [663cb536] FINISH, GetImageInfoVDSCommand, return: org.ovirt.engine.core.common.businessentities.DiskImage@717f9358, log id: 20d9fc65 >2012-11-08 08:24:54,137 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-12) [4ec16bb8] START, CreateVmVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=e228f541-76b7-4a9b-b313-a6bacc2b73b9, vm=org.ovirt.engine.core.common.businessentities.VM@5ba69521), log id: 1ab7c290 >2012-11-08 08:24:54,148 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8702-12) [4ec16bb8] START, CreateVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=e228f541-76b7-4a9b-b313-a6bacc2b73b9, vm=org.ovirt.engine.core.common.businessentities.VM@5ba69521), log id: 613dae15 >2012-11-08 08:24:54,156 INFO [org.ovirt.engine.core.bll.AddVmTemplateCommand] (ajp-/127.0.0.1:8702-2) [663cb536] Running command: AddVmTemplateCommand internal: false. Entities affected : ID: f4c98a0b-0578-485f-90f4-743df3d05d7e Type: StoragePool >2012-11-08 08:24:54,164 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (ajp-/127.0.0.1:8702-2) [663cb536] START, SetVmStatusVDSCommand( vmId = 3b031d10-a679-4211-88d6-6b31c6dd7537, status = ImageLocked), log id: 2bd0427b >2012-11-08 08:24:54,174 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (ajp-/127.0.0.1:8702-2) [663cb536] FINISH, SetVmStatusVDSCommand, log id: 2bd0427b >2012-11-08 08:24:54,187 INFO [org.ovirt.engine.core.bll.CreateImageTemplateCommand] (ajp-/127.0.0.1:8702-2) [3a8521c7] Running command: CreateImageTemplateCommand internal: true. Entities affected : ID: 00068b9a-c8e3-4e01-b2c4-963931bfae65 Type: Storage >2012-11-08 08:24:54,191 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (ajp-/127.0.0.1:8702-2) [3a8521c7] START, GetImageInfoVDSCommand( storagePoolId = f4c98a0b-0578-485f-90f4-743df3d05d7e, ignoreFailoverLimit = false, compatabilityVersion = null, storageDomainId = 00068b9a-c8e3-4e01-b2c4-963931bfae65, imageGroupId = 65c401eb-e12d-4d68-9fb3-e8237170875a, imageId = 13692da5-c6b6-494f-a94f-b295eeee4833), log id: 31ca8c27 >2012-11-08 08:24:54,199 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8702-12) [4ec16bb8] org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand spiceSslCipherSuite=DEFAULT,memSize=512,kvmEnable=true,smp=1,emulatedMachine=rhel6.3.0,vmType=kvm,keyboardLayout=en-us,nice=0,display=qxl,smpCoresPerSocket=1,spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard,timeOffset=-43200,transparentHugePages=true,vmId=e228f541-76b7-4a9b-b313-a6bacc2b73b9,devices=[Ljava.util.Map;@68b76dfa,acpiEnable=true,vmName=b4,cpuType=Nehalem,custom={} >2012-11-08 08:24:54,199 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8702-12) [4ec16bb8] FINISH, CreateVDSCommand, log id: 613dae15 >2012-11-08 08:24:54,200 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-12) [4ec16bb8] IncreasePendingVms::CreateVmIncreasing vds rhs-gp-srv11.lab.eng.blr.redhat.com pending vcpu count, now 1. Vm: b4 >2012-11-08 08:24:54,202 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-12) [4ec16bb8] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 1ab7c290 >2012-11-08 08:24:54,248 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (ajp-/127.0.0.1:8702-2) [3a8521c7] FINISH, GetImageInfoVDSCommand, return: org.ovirt.engine.core.common.businessentities.DiskImage@717f9358, log id: 31ca8c27 >2012-11-08 08:24:54,266 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CopyImageVDSCommand] (ajp-/127.0.0.1:8702-2) [3a8521c7] START, CopyImageVDSCommand( storagePoolId = f4c98a0b-0578-485f-90f4-743df3d05d7e, ignoreFailoverLimit = false, compatabilityVersion = 3.1, storageDomainId = 00068b9a-c8e3-4e01-b2c4-963931bfae65, imageGroupId = 65c401eb-e12d-4d68-9fb3-e8237170875a, imageId = 13692da5-c6b6-494f-a94f-b295eeee4833, dstImageGroupId = e95ccbb4-5049-43fd-b367-8658058f36ee, vmId = 3b031d10-a679-4211-88d6-6b31c6dd7537, dstImageId = 352c6d5b-ea13-4556-b24f-88bd89dd1fb6, imageDescription = Active VM, dstStorageDomainId = 00068b9a-c8e3-4e01-b2c4-963931bfae65, copyVolumeType = SharedVol, volumeFormat = RAW, preallocate = Sparse, postZero = false, force = false), log id: 747a8983 >2012-11-08 08:24:54,267 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CopyImageVDSCommand] (ajp-/127.0.0.1:8702-2) [3a8521c7] -- CopyImageVDSCommand::ExecuteIrsBrokerCommand: calling 'copyImage' with two new parameters: description and UUID >2012-11-08 08:24:54,267 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CopyImageVDSCommand] (ajp-/127.0.0.1:8702-2) [3a8521c7] -- copyImage parameters: > sdUUID=00068b9a-c8e3-4e01-b2c4-963931bfae65 > spUUID=f4c98a0b-0578-485f-90f4-743df3d05d7e > vmGUID=3b031d10-a679-4211-88d6-6b31c6dd7537 > srcImageGUID=65c401eb-e12d-4d68-9fb3-e8237170875a > srcVolUUID=13692da5-c6b6-494f-a94f-b295eeee4833 > dstImageGUID=e95ccbb4-5049-43fd-b367-8658058f36ee > dstVolUUID=352c6d5b-ea13-4556-b24f-88bd89dd1fb6 > descr=Active VM > >2012-11-08 08:24:54,450 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CopyImageVDSCommand] (ajp-/127.0.0.1:8702-2) [3a8521c7] FINISH, CopyImageVDSCommand, return: 00000000-0000-0000-0000-000000000000, log id: 747a8983 >2012-11-08 08:24:54,454 INFO [org.ovirt.engine.core.bll.EntityAsyncTask] (ajp-/127.0.0.1:8702-2) [3a8521c7] EntityAsyncTask::Adding EntityMultiAsyncTasks object for entity 54a99fbb-bb42-4f9f-acfd-d495dae03e3a >2012-11-08 08:24:54,454 INFO [org.ovirt.engine.core.bll.EntityMultiAsyncTasks] (ajp-/127.0.0.1:8702-2) [3a8521c7] EntityMultiAsyncTasks::AttachTask: Attaching task f5afe852-37d5-4c6a-84bd-867f469a6017 to entity 54a99fbb-bb42-4f9f-acfd-d495dae03e3a. >2012-11-08 08:24:54,455 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskUtils] (ajp-/127.0.0.1:8702-2) [3a8521c7] Adding task f5afe852-37d5-4c6a-84bd-867f469a6017 to DataBase >2012-11-08 08:24:54,466 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (ajp-/127.0.0.1:8702-2) [3a8521c7] Adding task f5afe852-37d5-4c6a-84bd-867f469a6017 (Parent Command AddVmTemplate, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters), polling hasn't started yet.. >2012-11-08 08:24:54,485 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (ajp-/127.0.0.1:8702-2) [3a8521c7] BaseAsyncTask::StartPollingTask: Starting to poll task f5afe852-37d5-4c6a-84bd-867f469a6017. >2012-11-08 08:24:54,485 INFO [org.ovirt.engine.core.bll.EntityMultiAsyncTasks] (ajp-/127.0.0.1:8702-2) [3a8521c7] EntityMultiAsyncTasks::StartPollingTask: Current Action Type for entity 54a99fbb-bb42-4f9f-acfd-d495dae03e3a is AddVmTemplate (determined by task f5afe852-37d5-4c6a-84bd-867f469a6017) >2012-11-08 08:24:55,797 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-52) VM b4 e228f541-76b7-4a9b-b313-a6bacc2b73b9 moved from WaitForLaunch --> PoweringUp >2012-11-08 08:24:55,797 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (QuartzScheduler_Worker-52) START, FullListVdsCommand( HostId = 96739be4-2990-11e2-8577-52540035ed48, vds=null, vmIds=[e228f541-76b7-4a9b-b313-a6bacc2b73b9]), log id: 140d4170 >2012-11-08 08:24:55,823 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (QuartzScheduler_Worker-52) FINISH, FullListVdsCommand, return: [Lorg.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcStruct;@2c9c1f11, log id: 140d4170 >2012-11-08 08:24:58,150 ERROR [org.ovirt.engine.core.bll.GetoVirtISOsQuery] (ajp-/127.0.0.1:8702-1) ovirt ISOs directory not found. Search in: /usr/share/rhev-hypervisor >2012-11-08 08:24:58,770 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-61) Polling and updating Async Tasks: 1 tasks, 1 tasks to poll now >2012-11-08 08:24:58,796 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (QuartzScheduler_Worker-61) SPMAsyncTask::PollTask: Polling task f5afe852-37d5-4c6a-84bd-867f469a6017 (Parent Command AddVmTemplate, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status running. >2012-11-08 08:24:58,796 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-61) Finished polling Tasks, will poll again in 10 seconds. >2012-11-08 08:25:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-64) Autorecovering hosts is disabled, skipping >2012-11-08 08:25:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-64) Autorecovering storage domains is disabled, skipping >2012-11-08 08:25:02,182 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-78) VM b1 67bd2d56-8e0b-4328-bcfd-ea7f4758c885 moved from RebootInProgress --> Up >2012-11-08 08:25:19,000 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-20) VM b3 7fa6f92f-755f-44c3-8f35-b4c56410c87b moved from RebootInProgress --> Up >2012-11-08 08:25:21,774 INFO [org.ovirt.engine.core.bll.SetVmTicketCommand] (ajp-/127.0.0.1:8702-2) [117419ad] Running command: SetVmTicketCommand internal: false. Entities affected : ID: e228f541-76b7-4a9b-b313-a6bacc2b73b9 Type: VM >2012-11-08 08:25:21,796 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp-/127.0.0.1:8702-2) [117419ad] START, SetVmTicketVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=e228f541-76b7-4a9b-b313-a6bacc2b73b9, ticket=HNKth2RZkFvd, validTime=120,m userName=admin@internal, userId=fdfc627c-d875-11e0-90f0-83df133b58cc), log id: 111ed41d >2012-11-08 08:25:21,826 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp-/127.0.0.1:8702-2) [117419ad] FINISH, SetVmTicketVDSCommand, log id: 111ed41d >2012-11-08 08:25:28,882 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (QuartzScheduler_Worker-49) SPMAsyncTask::PollTask: Polling task f5afe852-37d5-4c6a-84bd-867f469a6017 (Parent Command AddVmTemplate, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status finished, result 'success'. >2012-11-08 08:25:28,886 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (QuartzScheduler_Worker-49) BaseAsyncTask::OnTaskEndSuccess: Task f5afe852-37d5-4c6a-84bd-867f469a6017 (Parent Command AddVmTemplate, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended successfully. >2012-11-08 08:25:28,886 INFO [org.ovirt.engine.core.bll.EntityAsyncTask] (QuartzScheduler_Worker-49) EntityAsyncTask::EndActionIfNecessary: All tasks of entity 54a99fbb-bb42-4f9f-acfd-d495dae03e3a has ended -> executing EndAction >2012-11-08 08:25:28,887 INFO [org.ovirt.engine.core.bll.EntityAsyncTask] (QuartzScheduler_Worker-49) EntityAsyncTask::EndAction: Ending action for 1 tasks (entity ID: 54a99fbb-bb42-4f9f-acfd-d495dae03e3a): calling EndAction for action type AddVmTemplate. >2012-11-08 08:25:28,887 INFO [org.ovirt.engine.core.bll.EntityAsyncTask] (pool-4-thread-50) EntityAsyncTask::EndCommandAction [within thread]context: Attempting to EndAction AddVmTemplate >2012-11-08 08:25:28,907 INFO [org.ovirt.engine.core.bll.AddVmTemplateCommand] (pool-4-thread-50) [663cb536] Ending command successfully: org.ovirt.engine.core.bll.AddVmTemplateCommand >2012-11-08 08:25:28,911 INFO [org.ovirt.engine.core.bll.CreateImageTemplateCommand] (pool-4-thread-50) [3a8521c7] Ending command successfully: org.ovirt.engine.core.bll.CreateImageTemplateCommand >2012-11-08 08:25:28,919 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (pool-4-thread-50) [3a8521c7] START, GetImageInfoVDSCommand( storagePoolId = f4c98a0b-0578-485f-90f4-743df3d05d7e, ignoreFailoverLimit = false, compatabilityVersion = null, storageDomainId = 00068b9a-c8e3-4e01-b2c4-963931bfae65, imageGroupId = e95ccbb4-5049-43fd-b367-8658058f36ee, imageId = 352c6d5b-ea13-4556-b24f-88bd89dd1fb6), log id: 45abbf89 >2012-11-08 08:25:28,978 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (pool-4-thread-50) [3a8521c7] FINISH, GetImageInfoVDSCommand, return: org.ovirt.engine.core.common.businessentities.DiskImage@5dd1ba86, log id: 45abbf89 >2012-11-08 08:25:28,995 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UpdateVMVDSCommand] (pool-4-thread-50) START, UpdateVMVDSCommand( storagePoolId = f4c98a0b-0578-485f-90f4-743df3d05d7e, ignoreFailoverLimit = false, compatabilityVersion = null, storageDomainId = 00000000-0000-0000-0000-000000000000, infoDictionary.size = 1), log id: 1307860b >2012-11-08 08:25:29,026 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UpdateVMVDSCommand] (pool-4-thread-50) FINISH, UpdateVMVDSCommand, log id: 1307860b >2012-11-08 08:25:29,026 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (pool-4-thread-50) START, SetVmStatusVDSCommand( vmId = 3b031d10-a679-4211-88d6-6b31c6dd7537, status = Down), log id: 52f145ef >2012-11-08 08:25:29,030 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (pool-4-thread-50) FINISH, SetVmStatusVDSCommand, log id: 52f145ef >2012-11-08 08:25:29,050 INFO [org.ovirt.engine.core.bll.EntityAsyncTask] (pool-4-thread-50) EntityAsyncTask::HandleEndActionResult [within thread]: EndAction for action type AddVmTemplate completed, handling the result. >2012-11-08 08:25:29,050 INFO [org.ovirt.engine.core.bll.EntityAsyncTask] (pool-4-thread-50) EntityAsyncTask::HandleEndActionResult [within thread]: EndAction for action type AddVmTemplate succeeded, clearing tasks. >2012-11-08 08:25:29,052 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (pool-4-thread-50) SPMAsyncTask::ClearAsyncTask: Attempting to clear task f5afe852-37d5-4c6a-84bd-867f469a6017 >2012-11-08 08:25:29,052 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (pool-4-thread-50) START, SPMClearTaskVDSCommand( storagePoolId = f4c98a0b-0578-485f-90f4-743df3d05d7e, ignoreFailoverLimit = false, compatabilityVersion = null, taskId = f5afe852-37d5-4c6a-84bd-867f469a6017), log id: 6c1325fc >2012-11-08 08:25:29,054 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (pool-4-thread-50) START, HSMClearTaskVDSCommand(HostName = rhs-arch-srv2.lab.eng.blr.redhat.com, HostId = cf017816-29a2-11e2-949e-52540035ed48, taskId=f5afe852-37d5-4c6a-84bd-867f469a6017), log id: 45a995e7 >2012-11-08 08:25:29,102 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (pool-4-thread-50) FINISH, HSMClearTaskVDSCommand, log id: 45a995e7 >2012-11-08 08:25:29,102 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (pool-4-thread-50) FINISH, SPMClearTaskVDSCommand, log id: 6c1325fc >2012-11-08 08:25:29,104 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (pool-4-thread-50) BaseAsyncTask::RemoveTaskFromDB: Removed task f5afe852-37d5-4c6a-84bd-867f469a6017 from DataBase >2012-11-08 08:25:29,104 INFO [org.ovirt.engine.core.bll.EntityAsyncTask] (pool-4-thread-50) EntityAsyncTask::HandleEndActionResult [within thread]: Removing EntityMultiAsyncTasks object for entity 54a99fbb-bb42-4f9f-acfd-d495dae03e3a >2012-11-08 08:25:49,459 INFO [org.ovirt.engine.core.bll.SetVmTicketCommand] (ajp-/127.0.0.1:8702-8) [721b726c] Running command: SetVmTicketCommand internal: false. Entities affected : ID: e228f541-76b7-4a9b-b313-a6bacc2b73b9 Type: VM >2012-11-08 08:25:49,486 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp-/127.0.0.1:8702-8) [721b726c] START, SetVmTicketVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=e228f541-76b7-4a9b-b313-a6bacc2b73b9, ticket=5ndAUYDhQcdJ, validTime=120,m userName=admin@internal, userId=fdfc627c-d875-11e0-90f0-83df133b58cc), log id: 572b20ce >2012-11-08 08:25:49,516 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp-/127.0.0.1:8702-8) [721b726c] FINISH, SetVmTicketVDSCommand, log id: 572b20ce >2012-11-08 08:25:54,439 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-28) VM b4 e228f541-76b7-4a9b-b313-a6bacc2b73b9 moved from PoweringUp --> Up >2012-11-08 08:25:54,506 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsDynamicDataVDSCommand] (QuartzScheduler_Worker-28) START, UpdateVdsDynamicDataVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vdsDynamic=org.ovirt.engine.core.common.businessentities.VdsDynamic@59b4b72e), log id: 717ad32f >2012-11-08 08:25:54,508 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsDynamicDataVDSCommand] (QuartzScheduler_Worker-28) FINISH, UpdateVdsDynamicDataVDSCommand, log id: 717ad32f >2012-11-08 08:26:12,255 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsoPrefixVDSCommand] (ajp-/127.0.0.1:8702-16) [4115f986] START, IsoPrefixVDSCommand( storagePoolId = f4c98a0b-0578-485f-90f4-743df3d05d7e, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 7b536750 >2012-11-08 08:26:12,284 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsoPrefixVDSCommand] (ajp-/127.0.0.1:8702-16) [4115f986] FINISH, IsoPrefixVDSCommand, return: /rhev/data-center/f4c98a0b-0578-485f-90f4-743df3d05d7e/7a268ac7-857a-4c53-966d-247deb509161/images/11111111-1111-1111-1111-111111111111, log id: 7b536750 >2012-11-08 08:26:12,317 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-16) [4115f986] START, IsValidVDSCommand( storagePoolId = f4c98a0b-0578-485f-90f4-743df3d05d7e, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 634989d >2012-11-08 08:26:12,318 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-16) [4115f986] FINISH, IsValidVDSCommand, return: true, log id: 634989d >2012-11-08 08:26:12,334 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp-/127.0.0.1:8702-16) [4115f986] START, IsVmDuringInitiatingVDSCommand( vmId = 3b031d10-a679-4211-88d6-6b31c6dd7537), log id: 22824bee >2012-11-08 08:26:12,334 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp-/127.0.0.1:8702-16) [4115f986] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 22824bee >2012-11-08 08:26:12,370 INFO [org.ovirt.engine.core.bll.RunVmOnceCommand] (ajp-/127.0.0.1:8702-16) [4115f986] Running command: RunVmOnceCommand internal: false. Entities affected : ID: 3b031d10-a679-4211-88d6-6b31c6dd7537 Type: VM >2012-11-08 08:26:12,417 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-16) [4115f986] START, CreateVmVDSCommand(HostName = rhs-arch-srv2.lab.eng.blr.redhat.com, HostId = cf017816-29a2-11e2-949e-52540035ed48, vmId=3b031d10-a679-4211-88d6-6b31c6dd7537, vm=org.ovirt.engine.core.common.businessentities.VM@57b2989e), log id: 54ea55c5 >2012-11-08 08:26:12,422 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8702-16) [4115f986] START, CreateVDSCommand(HostName = rhs-arch-srv2.lab.eng.blr.redhat.com, HostId = cf017816-29a2-11e2-949e-52540035ed48, vmId=3b031d10-a679-4211-88d6-6b31c6dd7537, vm=org.ovirt.engine.core.common.businessentities.VM@57b2989e), log id: 344ac92f >2012-11-08 08:26:12,484 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8702-16) [4115f986] org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand spiceSslCipherSuite=DEFAULT,memSize=4200,kvmEnable=true,smp=2,emulatedMachine=rhel6.3.0,vmType=kvm,keyboardLayout=en-us,pitReinjection=false,nice=0,display=qxl,smpCoresPerSocket=1,spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard,timeOffset=-43200,transparentHugePages=true,vmId=3b031d10-a679-4211-88d6-6b31c6dd7537,devices=[Ljava.util.Map;@2ac0caa1,acpiEnable=true,vmName=FirstVM,cpuType=Nehalem,custom={} >2012-11-08 08:26:12,484 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8702-16) [4115f986] FINISH, CreateVDSCommand, log id: 344ac92f >2012-11-08 08:26:12,485 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-16) [4115f986] IncreasePendingVms::CreateVmIncreasing vds rhs-arch-srv2.lab.eng.blr.redhat.com pending vcpu count, now 2. Vm: FirstVM >2012-11-08 08:26:12,488 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-16) [4115f986] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 54ea55c5 >2012-11-08 08:26:15,887 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (QuartzScheduler_Worker-84) START, DestroyVDSCommand(HostName = rhs-arch-srv2.lab.eng.blr.redhat.com, HostId = cf017816-29a2-11e2-949e-52540035ed48, vmId=3b031d10-a679-4211-88d6-6b31c6dd7537, force=false, secondsToWait=0, gracefully=false), log id: 6df6dde8 >2012-11-08 08:26:16,273 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (QuartzScheduler_Worker-84) FINISH, DestroyVDSCommand, log id: 6df6dde8 >2012-11-08 08:26:16,282 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-84) Running on vds during rerun failed vm: null >2012-11-08 08:26:16,283 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-84) vm FirstVM running in db and not running in vds - add to rerun treatment. vds rhs-arch-srv2.lab.eng.blr.redhat.com >2012-11-08 08:26:16,287 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-84) Rerun vm 3b031d10-a679-4211-88d6-6b31c6dd7537. Called from vds rhs-arch-srv2.lab.eng.blr.redhat.com >2012-11-08 08:26:16,290 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsDynamicDataVDSCommand] (pool-4-thread-45) START, UpdateVdsDynamicDataVDSCommand(HostName = rhs-arch-srv2.lab.eng.blr.redhat.com, HostId = cf017816-29a2-11e2-949e-52540035ed48, vdsDynamic=org.ovirt.engine.core.common.businessentities.VdsDynamic@195dae16), log id: 7b3a3766 >2012-11-08 08:26:16,292 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsDynamicDataVDSCommand] (pool-4-thread-45) FINISH, UpdateVdsDynamicDataVDSCommand, log id: 7b3a3766 >2012-11-08 08:26:16,313 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (pool-4-thread-45) START, IsValidVDSCommand( storagePoolId = f4c98a0b-0578-485f-90f4-743df3d05d7e, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 598e3f93 >2012-11-08 08:26:16,313 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (pool-4-thread-45) FINISH, IsValidVDSCommand, return: true, log id: 598e3f93 >2012-11-08 08:26:16,323 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (pool-4-thread-45) START, IsVmDuringInitiatingVDSCommand( vmId = 3b031d10-a679-4211-88d6-6b31c6dd7537), log id: 60426131 >2012-11-08 08:26:16,323 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (pool-4-thread-45) FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 60426131 >2012-11-08 08:26:16,344 INFO [org.ovirt.engine.core.bll.VdsSelector] (pool-4-thread-45) VDS rhs-gp-srv11.lab.eng.blr.redhat.com 96739be4-2990-11e2-8577-52540035ed48 is not in up status or belongs to the VM's cluster VDS rhs-gp-srv4.lab.eng.blr.redhat.com d2bd32d4-2992-11e2-ab59-52540035ed48 is not in up status or belongs to the VM's cluster VDS rhs-gp-srv15.lab.eng.blr.redhat.com fca61124-29a6-11e2-a257-52540035ed48 is not in up status or belongs to the VM's cluster VDS rhs-gp-srv12.lab.eng.blr.redhat.com f047378c-29a6-11e2-8a79-52540035ed48 is not in up status or belongs to the VM's cluster VDS rhs-arch-srv2.lab.eng.blr.redhat.com cf017816-29a2-11e2-949e-52540035ed48 is the same host the VM is currently running on >2012-11-08 08:26:16,344 WARN [org.ovirt.engine.core.bll.RunVmOnceCommand] (pool-4-thread-45) CanDoAction of action RunVmOnce failed. Reasons:VAR__ACTION__RUN,VAR__TYPE__VM,$VmName FirstVM,VAR__ACTION__RUN,VAR__TYPE__VM,ACTION_TYPE_FAILED_VDS_VM_CLUSTER >2012-11-08 08:26:23,617 ERROR [org.ovirt.engine.core.bll.GetoVirtISOsQuery] (ajp-/127.0.0.1:8702-15) ovirt ISOs directory not found. Search in: /usr/share/rhev-hypervisor >2012-11-08 08:26:53,230 INFO [org.ovirt.engine.core.bll.RunVmCommand] (ajp-/127.0.0.1:8702-5) Lock Acquired to object EngineLock [exclusiveLocks= key: 3b031d10-a679-4211-88d6-6b31c6dd7537 value: VM >, sharedLocks= ] >2012-11-08 08:26:53,240 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-5) START, IsValidVDSCommand( storagePoolId = f4c98a0b-0578-485f-90f4-743df3d05d7e, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 670efc15 >2012-11-08 08:26:53,241 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-5) FINISH, IsValidVDSCommand, return: true, log id: 670efc15 >2012-11-08 08:26:53,253 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp-/127.0.0.1:8702-5) START, IsVmDuringInitiatingVDSCommand( vmId = 3b031d10-a679-4211-88d6-6b31c6dd7537), log id: 52fae10 >2012-11-08 08:26:53,253 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp-/127.0.0.1:8702-5) FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 52fae10 >2012-11-08 08:26:53,312 INFO [org.ovirt.engine.core.bll.RunVmCommand] (pool-4-thread-50) [69316685] Running command: RunVmCommand internal: false. Entities affected : ID: 3b031d10-a679-4211-88d6-6b31c6dd7537 Type: VM >2012-11-08 08:26:53,364 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsoPrefixVDSCommand] (pool-4-thread-50) [69316685] START, IsoPrefixVDSCommand( storagePoolId = f4c98a0b-0578-485f-90f4-743df3d05d7e, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 5b721fc0 >2012-11-08 08:26:53,364 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsoPrefixVDSCommand] (pool-4-thread-50) [69316685] FINISH, IsoPrefixVDSCommand, return: /rhev/data-center/f4c98a0b-0578-485f-90f4-743df3d05d7e/7a268ac7-857a-4c53-966d-247deb509161/images/11111111-1111-1111-1111-111111111111, log id: 5b721fc0 >2012-11-08 08:26:53,365 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (pool-4-thread-50) [69316685] START, CreateVmVDSCommand(HostName = rhs-arch-srv2.lab.eng.blr.redhat.com, HostId = cf017816-29a2-11e2-949e-52540035ed48, vmId=3b031d10-a679-4211-88d6-6b31c6dd7537, vm=org.ovirt.engine.core.common.businessentities.VM@491e8528), log id: 2182a527 >2012-11-08 08:26:53,371 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (pool-4-thread-50) [69316685] START, CreateVDSCommand(HostName = rhs-arch-srv2.lab.eng.blr.redhat.com, HostId = cf017816-29a2-11e2-949e-52540035ed48, vmId=3b031d10-a679-4211-88d6-6b31c6dd7537, vm=org.ovirt.engine.core.common.businessentities.VM@491e8528), log id: 223abe63 >2012-11-08 08:26:53,416 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (pool-4-thread-50) [69316685] org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand spiceSslCipherSuite=DEFAULT,memSize=4200,kvmEnable=true,smp=2,emulatedMachine=rhel6.3.0,vmType=kvm,keyboardLayout=en-us,pitReinjection=false,nice=0,display=qxl,smpCoresPerSocket=1,spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard,timeOffset=-43200,transparentHugePages=true,vmId=3b031d10-a679-4211-88d6-6b31c6dd7537,devices=[Ljava.util.Map;@49c222a7,acpiEnable=true,vmName=FirstVM,cpuType=Nehalem,custom={} >2012-11-08 08:26:53,416 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (pool-4-thread-50) [69316685] FINISH, CreateVDSCommand, log id: 223abe63 >2012-11-08 08:26:53,416 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (pool-4-thread-50) [69316685] IncreasePendingVms::CreateVmIncreasing vds rhs-arch-srv2.lab.eng.blr.redhat.com pending vcpu count, now 2. Vm: FirstVM >2012-11-08 08:26:53,420 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (pool-4-thread-50) [69316685] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 2182a527 >2012-11-08 08:26:53,422 INFO [org.ovirt.engine.core.bll.RunVmCommand] (pool-4-thread-50) [69316685] Lock freed to object EngineLock [exclusiveLocks= key: 3b031d10-a679-4211-88d6-6b31c6dd7537 value: VM >, sharedLocks= ] >2012-11-08 08:26:56,376 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-10) Setting new tasks map. The map contains now 0 tasks >2012-11-08 08:26:56,377 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-10) Cleared all tasks of pool f4c98a0b-0578-485f-90f4-743df3d05d7e. >2012-11-08 08:26:56,379 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-10) Could not find vds that is spm and non-operational. >2012-11-08 08:26:56,984 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (QuartzScheduler_Worker-12) START, DestroyVDSCommand(HostName = rhs-arch-srv2.lab.eng.blr.redhat.com, HostId = cf017816-29a2-11e2-949e-52540035ed48, vmId=3b031d10-a679-4211-88d6-6b31c6dd7537, force=false, secondsToWait=0, gracefully=false), log id: 3934d5bb >2012-11-08 08:26:57,332 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (QuartzScheduler_Worker-12) FINISH, DestroyVDSCommand, log id: 3934d5bb >2012-11-08 08:26:57,344 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-12) Running on vds during rerun failed vm: null >2012-11-08 08:26:57,345 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-12) vm FirstVM running in db and not running in vds - add to rerun treatment. vds rhs-arch-srv2.lab.eng.blr.redhat.com >2012-11-08 08:26:57,349 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-12) Rerun vm 3b031d10-a679-4211-88d6-6b31c6dd7537. Called from vds rhs-arch-srv2.lab.eng.blr.redhat.com >2012-11-08 08:26:57,351 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsDynamicDataVDSCommand] (pool-4-thread-50) START, UpdateVdsDynamicDataVDSCommand(HostName = rhs-arch-srv2.lab.eng.blr.redhat.com, HostId = cf017816-29a2-11e2-949e-52540035ed48, vdsDynamic=org.ovirt.engine.core.common.businessentities.VdsDynamic@195dae16), log id: 7600531b >2012-11-08 08:26:57,353 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsDynamicDataVDSCommand] (pool-4-thread-50) FINISH, UpdateVdsDynamicDataVDSCommand, log id: 7600531b >2012-11-08 08:26:57,361 INFO [org.ovirt.engine.core.bll.RunVmCommand] (pool-4-thread-50) Lock Acquired to object EngineLock [exclusiveLocks= key: 3b031d10-a679-4211-88d6-6b31c6dd7537 value: VM >, sharedLocks= ] >2012-11-08 08:26:57,365 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (pool-4-thread-50) START, IsValidVDSCommand( storagePoolId = f4c98a0b-0578-485f-90f4-743df3d05d7e, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 360ca0f >2012-11-08 08:26:57,366 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (pool-4-thread-50) FINISH, IsValidVDSCommand, return: true, log id: 360ca0f >2012-11-08 08:26:57,375 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (pool-4-thread-50) START, IsVmDuringInitiatingVDSCommand( vmId = 3b031d10-a679-4211-88d6-6b31c6dd7537), log id: 47652a58 >2012-11-08 08:26:57,375 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (pool-4-thread-50) FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 47652a58 >2012-11-08 08:26:57,397 INFO [org.ovirt.engine.core.bll.VdsSelector] (pool-4-thread-50) VDS rhs-gp-srv11.lab.eng.blr.redhat.com 96739be4-2990-11e2-8577-52540035ed48 is not in up status or belongs to the VM's cluster VDS rhs-gp-srv4.lab.eng.blr.redhat.com d2bd32d4-2992-11e2-ab59-52540035ed48 is not in up status or belongs to the VM's cluster VDS rhs-gp-srv15.lab.eng.blr.redhat.com fca61124-29a6-11e2-a257-52540035ed48 is not in up status or belongs to the VM's cluster VDS rhs-gp-srv12.lab.eng.blr.redhat.com f047378c-29a6-11e2-8a79-52540035ed48 is not in up status or belongs to the VM's cluster VDS rhs-arch-srv2.lab.eng.blr.redhat.com cf017816-29a2-11e2-949e-52540035ed48 is the same host the VM is currently running on >2012-11-08 08:26:57,397 WARN [org.ovirt.engine.core.bll.RunVmCommand] (pool-4-thread-50) CanDoAction of action RunVm failed. Reasons:VAR__ACTION__RUN,VAR__TYPE__VM,$VmName FirstVM,VAR__ACTION__RUN,VAR__TYPE__VM,VAR__ACTION__RUN,VAR__TYPE__VM,ACTION_TYPE_FAILED_VDS_VM_CLUSTER >2012-11-08 08:26:57,397 INFO [org.ovirt.engine.core.bll.RunVmCommand] (pool-4-thread-50) Lock freed to object EngineLock [exclusiveLocks= key: 3b031d10-a679-4211-88d6-6b31c6dd7537 value: VM >, sharedLocks= ] >2012-11-08 08:26:59,102 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-61) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='fence-agents' message='fence-agents-3.1.5-17.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:27:04,541 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (QuartzScheduler_Worker-36) START, DestroyVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=67bd2d56-8e0b-4328-bcfd-ea7f4758c885, force=false, secondsToWait=0, gracefully=false), log id: 291a71f4 >2012-11-08 08:27:04,558 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (QuartzScheduler_Worker-36) FINISH, DestroyVDSCommand, log id: 291a71f4 >2012-11-08 08:27:04,582 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-36) vm b1 running in db and not running in vds - add to rerun treatment. vds rhs-gp-srv11.lab.eng.blr.redhat.com >2012-11-08 08:27:09,277 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-69) Installation of 10.70.36.19. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='fence-agents' message='fence-agents-3.1.5-17.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:27:19,848 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-61) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='qemu-img' message='package qemu-img is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:27:29,933 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-69) Installation of 10.70.36.19. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='qemu-img' message='package qemu-img is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:27:33,809 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-61) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='seabios' message='seabios-0.6.1.2-19.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:27:33,816 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-5) [36934300] START, IsValidVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 7bec1a8e >2012-11-08 08:27:33,817 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-5) [36934300] FINISH, IsValidVDSCommand, return: true, log id: 7bec1a8e >2012-11-08 08:27:33,833 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp-/127.0.0.1:8702-5) [36934300] START, IsVmDuringInitiatingVDSCommand( vmId = 28f91cc7-0841-425d-a0fd-8ea1fd15c636), log id: 378c2326 >2012-11-08 08:27:33,833 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp-/127.0.0.1:8702-5) [36934300] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 378c2326 >2012-11-08 08:27:33,868 INFO [org.ovirt.engine.core.bll.RunVmOnceCommand] (ajp-/127.0.0.1:8702-5) [36934300] Running command: RunVmOnceCommand internal: false. Entities affected : ID: 28f91cc7-0841-425d-a0fd-8ea1fd15c636 Type: VM >2012-11-08 08:27:33,926 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsoPrefixVDSCommand] (ajp-/127.0.0.1:8702-5) [36934300] START, IsoPrefixVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 4ffd11fe >2012-11-08 08:27:33,926 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsoPrefixVDSCommand] (ajp-/127.0.0.1:8702-5) [36934300] FINISH, IsoPrefixVDSCommand, return: /rhev/data-center/3a17dcac-a6e3-4563-a4b0-f27d89b68dd6/7a268ac7-857a-4c53-966d-247deb509161/images/11111111-1111-1111-1111-111111111111, log id: 4ffd11fe >2012-11-08 08:27:33,946 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-5) [36934300] START, CreateVmVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=28f91cc7-0841-425d-a0fd-8ea1fd15c636, vm=org.ovirt.engine.core.common.businessentities.VM@1572a971), log id: 2e9d7da0 >2012-11-08 08:27:33,957 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8702-5) [36934300] START, CreateVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=28f91cc7-0841-425d-a0fd-8ea1fd15c636, vm=org.ovirt.engine.core.common.businessentities.VM@1572a971), log id: 99f613c >2012-11-08 08:27:34,003 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8702-5) [36934300] org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand spiceSslCipherSuite=DEFAULT,memSize=512,kvmEnable=true,smp=1,emulatedMachine=rhel6.3.0,vmType=kvm,keyboardLayout=en-us,nice=0,display=qxl,smpCoresPerSocket=1,spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard,timeOffset=-43200,transparentHugePages=true,vmId=28f91cc7-0841-425d-a0fd-8ea1fd15c636,devices=[Ljava.util.Map;@411bedf4,acpiEnable=true,vmName=disrep4,cpuType=Nehalem,custom={} >2012-11-08 08:27:34,003 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8702-5) [36934300] FINISH, CreateVDSCommand, log id: 99f613c >2012-11-08 08:27:34,006 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-5) [36934300] IncreasePendingVms::CreateVmIncreasing vds rhs-gp-srv11.lab.eng.blr.redhat.com pending vcpu count, now 1. Vm: disrep4 >2012-11-08 08:27:34,008 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-5) [36934300] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 2e9d7da0 >2012-11-08 08:27:37,544 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-26) VM disrep4 28f91cc7-0841-425d-a0fd-8ea1fd15c636 moved from WaitForLaunch --> PoweringUp >2012-11-08 08:27:37,547 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (QuartzScheduler_Worker-26) START, FullListVdsCommand( HostId = 96739be4-2990-11e2-8577-52540035ed48, vds=null, vmIds=[28f91cc7-0841-425d-a0fd-8ea1fd15c636]), log id: 8415109 >2012-11-08 08:27:37,568 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (QuartzScheduler_Worker-26) FINISH, FullListVdsCommand, return: [Lorg.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcStruct;@782f2a1a, log id: 8415109 >2012-11-08 08:27:42,461 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-69) Installation of 10.70.36.19. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='seabios' message='seabios-0.6.1.2-19.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:27:46,754 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-61) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='pixman' message='pixman-0.18.4-1.el6_0.1.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:27:56,014 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-69) Installation of 10.70.36.19. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='pixman' message='pixman-0.18.4-1.el6_0.1.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:28:15,933 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-61) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='spice-server' message='spice-server-0.10.1-10.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:28:18,277 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-61) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='libjpeg' message='libjpeg-6b-46.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:28:24,496 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-69) Installation of 10.70.36.19. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='spice-server' message='spice-server-0.10.1-10.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:28:26,854 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-69) Installation of 10.70.36.19. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='libjpeg' message='libjpeg-6b-46.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:28:30,705 INFO [org.ovirt.engine.core.bll.AddVmCommand] (ajp-/127.0.0.1:8702-4) [5b50310c] Lock Acquired to object EngineLock [exclusiveLocks= key: ms_auto_vm value: VM_NAME >, sharedLocks= ] >2012-11-08 08:28:30,725 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-4) [5b50310c] START, IsValidVDSCommand( storagePoolId = f4c98a0b-0578-485f-90f4-743df3d05d7e, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 43c09706 >2012-11-08 08:28:30,726 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-4) [5b50310c] FINISH, IsValidVDSCommand, return: true, log id: 43c09706 >2012-11-08 08:28:30,734 INFO [org.ovirt.engine.core.bll.AddVmCommand] (ajp-/127.0.0.1:8702-4) [5b50310c] Running command: AddVmCommand internal: false. Entities affected : ID: f9f22222-2976-11e2-a2cd-52540035ed48 Type: VdsGroups, ID: 54a99fbb-bb42-4f9f-acfd-d495dae03e3a Type: VmTemplate, ID: 00000000-0000-0000-0000-000000000000 Type: Storage >2012-11-08 08:28:30,735 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-4) [5b50310c] START, IsValidVDSCommand( storagePoolId = f4c98a0b-0578-485f-90f4-743df3d05d7e, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 9dc49d1 >2012-11-08 08:28:30,735 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-4) [5b50310c] FINISH, IsValidVDSCommand, return: true, log id: 9dc49d1 >2012-11-08 08:28:30,741 INFO [org.ovirt.engine.core.bll.MacPoolManager] (ajp-/127.0.0.1:8702-4) [5b50310c] MacPoolManager::allocateNewMac entered >2012-11-08 08:28:30,741 INFO [org.ovirt.engine.core.bll.MacPoolManager] (ajp-/127.0.0.1:8702-4) [5b50310c] MacPoolManager::allocateNewMac allocated mac = 00:1a:4a:46:22:60 >2012-11-08 08:28:30,744 INFO [org.ovirt.engine.core.bll.MacPoolManager] (ajp-/127.0.0.1:8702-4) [5b50310c] MacPoolManager::allocateNewMac entered >2012-11-08 08:28:30,744 INFO [org.ovirt.engine.core.bll.MacPoolManager] (ajp-/127.0.0.1:8702-4) [5b50310c] MacPoolManager::allocateNewMac allocated mac = 00:1a:4a:46:22:59 >2012-11-08 08:28:30,755 INFO [org.ovirt.engine.core.bll.AddVmCommand] (ajp-/127.0.0.1:8702-4) [5b50310c] Lock freed to object EngineLock [exclusiveLocks= key: ms_auto_vm value: VM_NAME >, sharedLocks= ] >2012-11-08 08:28:30,755 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (ajp-/127.0.0.1:8702-4) [5b50310c] START, SetVmStatusVDSCommand( vmId = a1059fbe-4eb3-493e-b8b6-b14ecba6fbe2, status = ImageLocked), log id: 789e4298 >2012-11-08 08:28:30,757 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (ajp-/127.0.0.1:8702-4) [5b50310c] FINISH, SetVmStatusVDSCommand, log id: 789e4298 >2012-11-08 08:28:30,805 INFO [org.ovirt.engine.core.bll.CreateSnapshotFromTemplateCommand] (ajp-/127.0.0.1:8702-4) [5afa5900] Running command: CreateSnapshotFromTemplateCommand internal: true. Entities affected : ID: 00068b9a-c8e3-4e01-b2c4-963931bfae65 Type: Storage >2012-11-08 08:28:30,813 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (ajp-/127.0.0.1:8702-4) [5afa5900] START, GetImageInfoVDSCommand( storagePoolId = f4c98a0b-0578-485f-90f4-743df3d05d7e, ignoreFailoverLimit = false, compatabilityVersion = null, storageDomainId = 00068b9a-c8e3-4e01-b2c4-963931bfae65, imageGroupId = e95ccbb4-5049-43fd-b367-8658058f36ee, imageId = 352c6d5b-ea13-4556-b24f-88bd89dd1fb6), log id: 75c9ad20 >2012-11-08 08:28:30,887 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (ajp-/127.0.0.1:8702-4) [5afa5900] FINISH, GetImageInfoVDSCommand, return: org.ovirt.engine.core.common.businessentities.DiskImage@5dd1ba86, log id: 75c9ad20 >2012-11-08 08:28:30,890 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateSnapshotVDSCommand] (ajp-/127.0.0.1:8702-4) [5afa5900] START, CreateSnapshotVDSCommand( storagePoolId = f4c98a0b-0578-485f-90f4-743df3d05d7e, ignoreFailoverLimit = false, compatabilityVersion = 3.1, storageDomainId = 00068b9a-c8e3-4e01-b2c4-963931bfae65, imageGroupId = 1d59d61e-5a79-471a-b722-4926b9866b9a, imageSizeInBytes = 10737418240, volumeFormat = COW, newImageId = 30d6e8e5-6ec7-40d7-880f-9d41faa51e94, newImageDescription = , imageId = 352c6d5b-ea13-4556-b24f-88bd89dd1fb6, sourceImageGroupId = e95ccbb4-5049-43fd-b367-8658058f36ee), log id: 3e7dc1f1 >2012-11-08 08:28:30,890 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateSnapshotVDSCommand] (ajp-/127.0.0.1:8702-4) [5afa5900] -- CreateSnapshotVDSCommand::ExecuteIrsBrokerCommand: calling 'createVolume' with two new parameters: description and UUID >2012-11-08 08:28:30,890 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateSnapshotVDSCommand] (ajp-/127.0.0.1:8702-4) [5afa5900] -- createVolume parameters: > sdUUID=00068b9a-c8e3-4e01-b2c4-963931bfae65 > spUUID=f4c98a0b-0578-485f-90f4-743df3d05d7e > imgGUID=1d59d61e-5a79-471a-b722-4926b9866b9a > size=10,737,418,240 bytes > volFormat=COW > volType=Sparse > volUUID=30d6e8e5-6ec7-40d7-880f-9d41faa51e94 > descr= > srcImgGUID=e95ccbb4-5049-43fd-b367-8658058f36ee > srcVolUUID=352c6d5b-ea13-4556-b24f-88bd89dd1fb6 > >2012-11-08 08:28:31,019 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateSnapshotVDSCommand] (ajp-/127.0.0.1:8702-4) [5afa5900] FINISH, CreateSnapshotVDSCommand, return: 30d6e8e5-6ec7-40d7-880f-9d41faa51e94, log id: 3e7dc1f1 >2012-11-08 08:28:31,022 INFO [org.ovirt.engine.core.bll.EntityAsyncTask] (ajp-/127.0.0.1:8702-4) [5afa5900] EntityAsyncTask::Adding EntityMultiAsyncTasks object for entity a1059fbe-4eb3-493e-b8b6-b14ecba6fbe2 >2012-11-08 08:28:31,022 INFO [org.ovirt.engine.core.bll.EntityMultiAsyncTasks] (ajp-/127.0.0.1:8702-4) [5afa5900] EntityMultiAsyncTasks::AttachTask: Attaching task 029f59ab-f8f3-4709-adbe-e810056bb81d to entity a1059fbe-4eb3-493e-b8b6-b14ecba6fbe2. >2012-11-08 08:28:31,023 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskUtils] (ajp-/127.0.0.1:8702-4) [5afa5900] Adding task 029f59ab-f8f3-4709-adbe-e810056bb81d to DataBase >2012-11-08 08:28:31,051 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (ajp-/127.0.0.1:8702-4) [5afa5900] Adding task 029f59ab-f8f3-4709-adbe-e810056bb81d (Parent Command AddVm, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters), polling hasn't started yet.. >2012-11-08 08:28:31,109 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (ajp-/127.0.0.1:8702-4) [5afa5900] BaseAsyncTask::StartPollingTask: Starting to poll task 029f59ab-f8f3-4709-adbe-e810056bb81d. >2012-11-08 08:28:31,110 INFO [org.ovirt.engine.core.bll.EntityMultiAsyncTasks] (ajp-/127.0.0.1:8702-4) [5afa5900] EntityMultiAsyncTasks::StartPollingTask: Current Action Type for entity a1059fbe-4eb3-493e-b8b6-b14ecba6fbe2 is AddVm (determined by task 029f59ab-f8f3-4709-adbe-e810056bb81d) >2012-11-08 08:28:35,805 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-99) VM disrep4 28f91cc7-0841-425d-a0fd-8ea1fd15c636 moved from PoweringUp --> Up >2012-11-08 08:28:35,837 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsDynamicDataVDSCommand] (QuartzScheduler_Worker-99) START, UpdateVdsDynamicDataVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vdsDynamic=org.ovirt.engine.core.common.businessentities.VdsDynamic@64318dd8), log id: 7bbe5160 >2012-11-08 08:28:35,839 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsDynamicDataVDSCommand] (QuartzScheduler_Worker-99) FINISH, UpdateVdsDynamicDataVDSCommand, log id: 7bbe5160 >2012-11-08 08:28:38,891 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-10) Polling and updating Async Tasks: 1 tasks, 1 tasks to poll now >2012-11-08 08:28:38,916 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (QuartzScheduler_Worker-10) SPMAsyncTask::PollTask: Polling task 029f59ab-f8f3-4709-adbe-e810056bb81d (Parent Command AddVm, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status finished, result 'success'. >2012-11-08 08:28:38,920 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (QuartzScheduler_Worker-10) BaseAsyncTask::OnTaskEndSuccess: Task 029f59ab-f8f3-4709-adbe-e810056bb81d (Parent Command AddVm, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended successfully. >2012-11-08 08:28:38,920 INFO [org.ovirt.engine.core.bll.EntityAsyncTask] (QuartzScheduler_Worker-10) EntityAsyncTask::EndActionIfNecessary: All tasks of entity a1059fbe-4eb3-493e-b8b6-b14ecba6fbe2 has ended -> executing EndAction >2012-11-08 08:28:38,920 INFO [org.ovirt.engine.core.bll.EntityAsyncTask] (QuartzScheduler_Worker-10) EntityAsyncTask::EndAction: Ending action for 1 tasks (entity ID: a1059fbe-4eb3-493e-b8b6-b14ecba6fbe2): calling EndAction for action type AddVm. >2012-11-08 08:28:38,920 INFO [org.ovirt.engine.core.bll.EntityAsyncTask] (pool-4-thread-50) EntityAsyncTask::EndCommandAction [within thread]context: Attempting to EndAction AddVm >2012-11-08 08:28:38,939 INFO [org.ovirt.engine.core.bll.AddVmCommand] (pool-4-thread-50) [5b50310c] Ending command successfully: org.ovirt.engine.core.bll.AddVmCommand >2012-11-08 08:28:38,939 INFO [org.ovirt.engine.core.bll.CreateSnapshotFromTemplateCommand] (pool-4-thread-50) [5afa5900] Ending command successfully: org.ovirt.engine.core.bll.CreateSnapshotFromTemplateCommand >2012-11-08 08:28:38,949 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (pool-4-thread-50) [5afa5900] START, GetImageInfoVDSCommand( storagePoolId = f4c98a0b-0578-485f-90f4-743df3d05d7e, ignoreFailoverLimit = false, compatabilityVersion = null, storageDomainId = 00068b9a-c8e3-4e01-b2c4-963931bfae65, imageGroupId = 1d59d61e-5a79-471a-b722-4926b9866b9a, imageId = 30d6e8e5-6ec7-40d7-880f-9d41faa51e94), log id: 7de801a3 >2012-11-08 08:28:39,011 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (pool-4-thread-50) [5afa5900] FINISH, GetImageInfoVDSCommand, return: org.ovirt.engine.core.common.businessentities.DiskImage@889d44b7, log id: 7de801a3 >2012-11-08 08:28:39,019 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (pool-4-thread-50) START, SetVmStatusVDSCommand( vmId = a1059fbe-4eb3-493e-b8b6-b14ecba6fbe2, status = Down), log id: 52d048dc >2012-11-08 08:28:39,022 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (pool-4-thread-50) FINISH, SetVmStatusVDSCommand, log id: 52d048dc >2012-11-08 08:28:39,038 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UpdateVMVDSCommand] (pool-4-thread-50) START, UpdateVMVDSCommand( storagePoolId = f4c98a0b-0578-485f-90f4-743df3d05d7e, ignoreFailoverLimit = false, compatabilityVersion = null, storageDomainId = 00000000-0000-0000-0000-000000000000, infoDictionary.size = 1), log id: 519183 >2012-11-08 08:28:39,068 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UpdateVMVDSCommand] (pool-4-thread-50) FINISH, UpdateVMVDSCommand, log id: 519183 >2012-11-08 08:28:39,085 INFO [org.ovirt.engine.core.bll.EntityAsyncTask] (pool-4-thread-50) EntityAsyncTask::HandleEndActionResult [within thread]: EndAction for action type AddVm completed, handling the result. >2012-11-08 08:28:39,085 INFO [org.ovirt.engine.core.bll.EntityAsyncTask] (pool-4-thread-50) EntityAsyncTask::HandleEndActionResult [within thread]: EndAction for action type AddVm succeeded, clearing tasks. >2012-11-08 08:28:39,088 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (pool-4-thread-50) SPMAsyncTask::ClearAsyncTask: Attempting to clear task 029f59ab-f8f3-4709-adbe-e810056bb81d >2012-11-08 08:28:39,088 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (pool-4-thread-50) START, SPMClearTaskVDSCommand( storagePoolId = f4c98a0b-0578-485f-90f4-743df3d05d7e, ignoreFailoverLimit = false, compatabilityVersion = null, taskId = 029f59ab-f8f3-4709-adbe-e810056bb81d), log id: 6f797449 >2012-11-08 08:28:39,090 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (pool-4-thread-50) START, HSMClearTaskVDSCommand(HostName = rhs-arch-srv2.lab.eng.blr.redhat.com, HostId = cf017816-29a2-11e2-949e-52540035ed48, taskId=029f59ab-f8f3-4709-adbe-e810056bb81d), log id: 701e3501 >2012-11-08 08:28:39,131 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (pool-4-thread-50) FINISH, HSMClearTaskVDSCommand, log id: 701e3501 >2012-11-08 08:28:39,131 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (pool-4-thread-50) FINISH, SPMClearTaskVDSCommand, log id: 6f797449 >2012-11-08 08:28:39,133 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (pool-4-thread-50) BaseAsyncTask::RemoveTaskFromDB: Removed task 029f59ab-f8f3-4709-adbe-e810056bb81d from DataBase >2012-11-08 08:28:39,133 INFO [org.ovirt.engine.core.bll.EntityAsyncTask] (pool-4-thread-50) EntityAsyncTask::HandleEndActionResult [within thread]: Removing EntityMultiAsyncTasks object for entity a1059fbe-4eb3-493e-b8b6-b14ecba6fbe2 >2012-11-08 08:28:46,298 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsoPrefixVDSCommand] (ajp-/127.0.0.1:8702-5) [6c9674f8] START, IsoPrefixVDSCommand( storagePoolId = f4c98a0b-0578-485f-90f4-743df3d05d7e, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 7ed4f58a >2012-11-08 08:28:46,298 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsoPrefixVDSCommand] (ajp-/127.0.0.1:8702-5) [6c9674f8] FINISH, IsoPrefixVDSCommand, return: /rhev/data-center/f4c98a0b-0578-485f-90f4-743df3d05d7e/7a268ac7-857a-4c53-966d-247deb509161/images/11111111-1111-1111-1111-111111111111, log id: 7ed4f58a >2012-11-08 08:28:46,334 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-5) [6c9674f8] START, IsValidVDSCommand( storagePoolId = f4c98a0b-0578-485f-90f4-743df3d05d7e, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 39bab4e3 >2012-11-08 08:28:46,338 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-5) [6c9674f8] FINISH, IsValidVDSCommand, return: true, log id: 39bab4e3 >2012-11-08 08:28:46,352 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp-/127.0.0.1:8702-5) [6c9674f8] START, IsVmDuringInitiatingVDSCommand( vmId = 3b031d10-a679-4211-88d6-6b31c6dd7537), log id: 358383ab >2012-11-08 08:28:46,353 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp-/127.0.0.1:8702-5) [6c9674f8] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 358383ab >2012-11-08 08:28:46,391 INFO [org.ovirt.engine.core.bll.RunVmOnceCommand] (ajp-/127.0.0.1:8702-5) [6c9674f8] Running command: RunVmOnceCommand internal: false. Entities affected : ID: 3b031d10-a679-4211-88d6-6b31c6dd7537 Type: VM >2012-11-08 08:28:46,428 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-5) [6c9674f8] START, CreateVmVDSCommand(HostName = rhs-arch-srv2.lab.eng.blr.redhat.com, HostId = cf017816-29a2-11e2-949e-52540035ed48, vmId=3b031d10-a679-4211-88d6-6b31c6dd7537, vm=org.ovirt.engine.core.common.businessentities.VM@36feaee3), log id: 7f76695f >2012-11-08 08:28:46,434 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8702-5) [6c9674f8] START, CreateVDSCommand(HostName = rhs-arch-srv2.lab.eng.blr.redhat.com, HostId = cf017816-29a2-11e2-949e-52540035ed48, vmId=3b031d10-a679-4211-88d6-6b31c6dd7537, vm=org.ovirt.engine.core.common.businessentities.VM@36feaee3), log id: 6ea7d310 >2012-11-08 08:28:46,490 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8702-5) [6c9674f8] org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand spiceSslCipherSuite=DEFAULT,memSize=4200,kvmEnable=true,smp=2,emulatedMachine=rhel6.3.0,vmType=kvm,keyboardLayout=en-us,pitReinjection=false,nice=0,display=qxl,smpCoresPerSocket=1,spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard,timeOffset=-43200,transparentHugePages=true,vmId=3b031d10-a679-4211-88d6-6b31c6dd7537,devices=[Ljava.util.Map;@76fd71ff,acpiEnable=true,vmName=FirstVM,cpuType=Nehalem,custom={} >2012-11-08 08:28:46,490 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8702-5) [6c9674f8] FINISH, CreateVDSCommand, log id: 6ea7d310 >2012-11-08 08:28:46,491 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-5) [6c9674f8] IncreasePendingVms::CreateVmIncreasing vds rhs-arch-srv2.lab.eng.blr.redhat.com pending vcpu count, now 2. Vm: FirstVM >2012-11-08 08:28:46,494 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-5) [6c9674f8] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 7f76695f >2012-11-08 08:28:49,196 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-32) VM FirstVM 3b031d10-a679-4211-88d6-6b31c6dd7537 moved from WaitForLaunch --> PoweringUp >2012-11-08 08:28:49,196 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (QuartzScheduler_Worker-32) START, FullListVdsCommand( HostId = cf017816-29a2-11e2-949e-52540035ed48, vds=null, vmIds=[3b031d10-a679-4211-88d6-6b31c6dd7537]), log id: 1c218bff >2012-11-08 08:28:49,224 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (QuartzScheduler_Worker-32) FINISH, FullListVdsCommand, return: [Lorg.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcStruct;@2b098619, log id: 1c218bff >2012-11-08 08:29:01,373 INFO [org.ovirt.engine.core.bll.SetVmTicketCommand] (ajp-/127.0.0.1:8702-8) [7de861e4] Running command: SetVmTicketCommand internal: false. Entities affected : ID: 3b031d10-a679-4211-88d6-6b31c6dd7537 Type: VM >2012-11-08 08:29:01,376 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp-/127.0.0.1:8702-8) [7de861e4] START, SetVmTicketVDSCommand(HostName = rhs-arch-srv2.lab.eng.blr.redhat.com, HostId = cf017816-29a2-11e2-949e-52540035ed48, vmId=3b031d10-a679-4211-88d6-6b31c6dd7537, ticket=Xz3kOxSY4ejl, validTime=120,m userName=admin@internal, userId=fdfc627c-d875-11e0-90f0-83df133b58cc), log id: 6df81e3b >2012-11-08 08:29:01,407 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp-/127.0.0.1:8702-8) [7de861e4] FINISH, SetVmTicketVDSCommand, log id: 6df81e3b >2012-11-08 08:29:35,063 ERROR [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-47) [225e1867] Installation of 10.70.36.16. Error: Repository JBOSS is listed more than once in the configuration >Repository qa-latest-5 is listed more than once in the configuration >Repository rhel-source is listed more than once in the configuration >Repository rhel-source-beta is listed more than once in the configuration >Repository rhs-client is listed more than once in the configuration >Repository JBOSS is listed more than once in the configuration >Repository qa-latest-5 is listed more than once in the configuration >Repository rhel-source is listed more than once in the configuration >Repository rhel-source-beta is listed more than once in the configuration >Repository rhs-client is listed more than once in the configuration >. (Stage: Running first installation script on Host) >2012-11-08 08:29:35,063 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-47) [225e1867] VDS message: Repository JBOSS is listed more than once in the configuration >2012-11-08 08:29:35,063 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-47) [225e1867] VDS message: Repository qa-latest-5 is listed more than once in the configuration >2012-11-08 08:29:35,063 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-47) [225e1867] VDS message: Repository rhel-source is listed more than once in the configuration >2012-11-08 08:29:35,063 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-47) [225e1867] VDS message: Repository rhel-source-beta is listed more than once in the configuration >2012-11-08 08:29:35,063 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-47) [225e1867] VDS message: Repository rhs-client is listed more than once in the configuration >2012-11-08 08:29:35,063 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-47) [225e1867] VDS message: Repository JBOSS is listed more than once in the configuration >2012-11-08 08:29:35,064 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-47) [225e1867] VDS message: Repository qa-latest-5 is listed more than once in the configuration >2012-11-08 08:29:35,064 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-47) [225e1867] VDS message: Repository rhel-source is listed more than once in the configuration >2012-11-08 08:29:35,064 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-47) [225e1867] VDS message: Repository rhel-source-beta is listed more than once in the configuration >2012-11-08 08:29:35,064 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-47) [225e1867] VDS message: Repository rhs-client is listed more than once in the configuration >2012-11-08 08:29:35,064 ERROR [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-4-thread-47) SSH error running command 10.70.36.16:'umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar -C "${MYTMP}" --no-same-permissions -o -x && "${MYTMP}"/setup -c 'ssl=true;management_port=54321' -O 'lab.eng.blr.redhat.com' -t 2012-11-08T13:19:35 -S /tmp/ovirt-id_rsa_89746fd4-ead3-4285-a67a-f7a25c5d9799 -p 80 -b -B rhevm http://guido.lab.eng.blr.redhat.com:80/Components/vds/ http://guido.lab.eng.blr.redhat.com:80/Components/vds/ 10.70.36.16 89746fd4-ead3-4285-a67a-f7a25c5d9799 False': javax.naming.TimeLimitExceededException: SSH session hard timeout host '10.70.36.16:22' > at org.ovirt.engine.core.utils.ssh.SSHClient.executeCommand(SSHClient.java:402) [engine-utils.jar:] > at org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH.executeCommand(VdsInstallerSSH.java:387) [engine-utils.jar:] > at org.ovirt.engine.core.bll.VdsInstaller.runBootstrapCommand(VdsInstaller.java:217) [engine-bll.jar:] > at org.ovirt.engine.core.bll.VdsInstaller.RunStage(VdsInstaller.java:341) [engine-bll.jar:] > at org.ovirt.engine.core.bll.VdsInstaller.Install(VdsInstaller.java:280) [engine-bll.jar:] > at org.ovirt.engine.core.bll.InstallVdsCommand.executeCommand(InstallVdsCommand.java:110) [engine-bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:825) [engine-bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:916) [engine-bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1300) [engine-bll.jar:] > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:168) [engine-utils.jar:] > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:107) [engine-utils.jar:] > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:931) [engine-bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:285) [engine-bll.jar:] > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:336) [engine-bll.jar:] > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:593) [engine-bll.jar:] > at sun.reflect.GeneratedMethodAccessor136.invoke(Unknown Source) [:1.7.0_09-icedtea] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_09-icedtea] > at java.lang.reflect.Method.invoke(Method.java:601) [rt.jar:1.7.0_09-icedtea] > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) [jboss-as-ee.jar:7.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:36) [jboss-as-ee.jar:7.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) [jboss-as-ejb3.jar:7.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) [jboss-as-ee.jar:7.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:53) [jboss-as-ejb3.jar:7.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:210) [jboss-as-ejb3.jar:7.1.2.Final-redhat-1] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:362) [jboss-as-ejb3.jar:7.1.2.Final-redhat-1] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:193) [jboss-as-ejb3.jar:7.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [jboss-as-ejb3.jar:7.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:42) [jboss-as-ejb3.jar:7.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) [jboss-as-ejb3.jar:7.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) [jboss-as-ee.jar:7.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) [jboss-as-ee.jar:7.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) [jboss-as-ee.jar:7.1.2.Final-redhat-1] > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:176) [jboss-as-ee.jar:7.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) [jboss-as-ee.jar:7.1.2.Final-redhat-1] > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) [engine-bll.jar:] > at org.ovirt.engine.core.bll.AddVdsCommand$3.run(AddVdsCommand.java:156) [engine-bll.jar:] > at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalWrapperRunnable.run(ThreadPoolUtil.java:64) [engine-utils.jar:] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_09-icedtea] > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) [rt.jar:1.7.0_09-icedtea] > at java.util.concurrent.FutureTask.run(FutureTask.java:166) [rt.jar:1.7.0_09-icedtea] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [rt.jar:1.7.0_09-icedtea] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [rt.jar:1.7.0_09-icedtea] > at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_09-icedtea] > >2012-11-08 08:29:35,074 ERROR [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-47) [225e1867] Installation of 10.70.36.16 has failed. Failure details: SSH command failed while executing at host '10.70.36.16', refer to logs for further information. (Stage: Running first installation script on Host) >2012-11-08 08:29:35,074 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-47) [225e1867] VDS message: SSH command failed while executing at host '10.70.36.16', refer to logs for further information >2012-11-08 08:29:35,074 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-47) [225e1867] Script ended, result is {1} >2012-11-08 08:29:35,076 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-4-thread-47) [225e1867] After Installation pool-4-thread-47 >2012-11-08 08:29:35,080 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-47) [225e1867] START, SetVdsStatusVDSCommand(HostName = rhs-gp-srv12.lab.eng.blr.redhat.com, HostId = f047378c-29a6-11e2-8a79-52540035ed48, status=InstallFailed, nonOperationalReason=NONE), log id: 2e4743d5 >2012-11-08 08:29:35,087 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-47) [225e1867] FINISH, SetVdsStatusVDSCommand, log id: 2e4743d5 >2012-11-08 08:29:46,912 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-10) VM FirstVM 3b031d10-a679-4211-88d6-6b31c6dd7537 moved from PoweringUp --> Up >2012-11-08 08:29:46,919 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsDynamicDataVDSCommand] (QuartzScheduler_Worker-10) START, UpdateVdsDynamicDataVDSCommand(HostName = rhs-arch-srv2.lab.eng.blr.redhat.com, HostId = cf017816-29a2-11e2-949e-52540035ed48, vdsDynamic=org.ovirt.engine.core.common.businessentities.VdsDynamic@c92c456d), log id: eb7ded6 >2012-11-08 08:29:46,920 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsDynamicDataVDSCommand] (QuartzScheduler_Worker-10) FINISH, UpdateVdsDynamicDataVDSCommand, log id: eb7ded6 >2012-11-08 08:29:55,810 ERROR [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-49) [a707348] Installation of 10.70.36.19. Error: Repository JBOSS is listed more than once in the configuration >Repository qa-latest-5 is listed more than once in the configuration >Repository rhel-source is listed more than once in the configuration >Repository rhel-source-beta is listed more than once in the configuration >Repository rhs-client is listed more than once in the configuration >Repository JBOSS is listed more than once in the configuration >Repository qa-latest-5 is listed more than once in the configuration >Repository rhel-source is listed more than once in the configuration >Repository rhel-source-beta is listed more than once in the configuration >Repository rhs-client is listed more than once in the configuration >. (Stage: Running first installation script on Host) >2012-11-08 08:29:55,810 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-49) [a707348] VDS message: Repository JBOSS is listed more than once in the configuration >2012-11-08 08:29:55,810 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-49) [a707348] VDS message: Repository qa-latest-5 is listed more than once in the configuration >2012-11-08 08:29:55,810 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-49) [a707348] VDS message: Repository rhel-source is listed more than once in the configuration >2012-11-08 08:29:55,810 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-49) [a707348] VDS message: Repository rhel-source-beta is listed more than once in the configuration >2012-11-08 08:29:55,810 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-49) [a707348] VDS message: Repository rhs-client is listed more than once in the configuration >2012-11-08 08:29:55,810 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-49) [a707348] VDS message: Repository JBOSS is listed more than once in the configuration >2012-11-08 08:29:55,810 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-49) [a707348] VDS message: Repository qa-latest-5 is listed more than once in the configuration >2012-11-08 08:29:55,810 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-49) [a707348] VDS message: Repository rhel-source is listed more than once in the configuration >2012-11-08 08:29:55,810 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-49) [a707348] VDS message: Repository rhel-source-beta is listed more than once in the configuration >2012-11-08 08:29:55,810 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-49) [a707348] VDS message: Repository rhs-client is listed more than once in the configuration >2012-11-08 08:29:55,810 ERROR [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-4-thread-49) SSH error running command 10.70.36.19:'umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar -C "${MYTMP}" --no-same-permissions -o -x && "${MYTMP}"/setup -c 'ssl=true;management_port=54321' -O 'lab.eng.blr.redhat.com' -t 2012-11-08T13:19:55 -S /tmp/ovirt-id_rsa_d3a65d1e-e54a-4810-890e-6cbdef7fdd9a -p 80 -b -B rhevm http://guido.lab.eng.blr.redhat.com:80/Components/vds/ http://guido.lab.eng.blr.redhat.com:80/Components/vds/ 10.70.36.19 d3a65d1e-e54a-4810-890e-6cbdef7fdd9a False': javax.naming.TimeLimitExceededException: SSH session hard timeout host '10.70.36.19:22' > at org.ovirt.engine.core.utils.ssh.SSHClient.executeCommand(SSHClient.java:402) [engine-utils.jar:] > at org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH.executeCommand(VdsInstallerSSH.java:387) [engine-utils.jar:] > at org.ovirt.engine.core.bll.VdsInstaller.runBootstrapCommand(VdsInstaller.java:217) [engine-bll.jar:] > at org.ovirt.engine.core.bll.VdsInstaller.RunStage(VdsInstaller.java:341) [engine-bll.jar:] > at org.ovirt.engine.core.bll.VdsInstaller.Install(VdsInstaller.java:280) [engine-bll.jar:] > at org.ovirt.engine.core.bll.InstallVdsCommand.executeCommand(InstallVdsCommand.java:110) [engine-bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:825) [engine-bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:916) [engine-bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1300) [engine-bll.jar:] > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:168) [engine-utils.jar:] > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:107) [engine-utils.jar:] > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:931) [engine-bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:285) [engine-bll.jar:] > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:336) [engine-bll.jar:] > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:593) [engine-bll.jar:] > at sun.reflect.GeneratedMethodAccessor136.invoke(Unknown Source) [:1.7.0_09-icedtea] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_09-icedtea] > at java.lang.reflect.Method.invoke(Method.java:601) [rt.jar:1.7.0_09-icedtea] > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) [jboss-as-ee.jar:7.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:36) [jboss-as-ee.jar:7.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) [jboss-as-ejb3.jar:7.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) [jboss-as-ee.jar:7.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:53) [jboss-as-ejb3.jar:7.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:210) [jboss-as-ejb3.jar:7.1.2.Final-redhat-1] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:362) [jboss-as-ejb3.jar:7.1.2.Final-redhat-1] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:193) [jboss-as-ejb3.jar:7.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [jboss-as-ejb3.jar:7.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:42) [jboss-as-ejb3.jar:7.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) [jboss-as-ejb3.jar:7.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) [jboss-as-ee.jar:7.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) [jboss-as-ee.jar:7.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) [jboss-as-ee.jar:7.1.2.Final-redhat-1] > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:176) [jboss-as-ee.jar:7.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) [jboss-as-ee.jar:7.1.2.Final-redhat-1] > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) [engine-bll.jar:] > at org.ovirt.engine.core.bll.AddVdsCommand$3.run(AddVdsCommand.java:156) [engine-bll.jar:] > at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalWrapperRunnable.run(ThreadPoolUtil.java:64) [engine-utils.jar:] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_09-icedtea] > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) [rt.jar:1.7.0_09-icedtea] > at java.util.concurrent.FutureTask.run(FutureTask.java:166) [rt.jar:1.7.0_09-icedtea] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [rt.jar:1.7.0_09-icedtea] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [rt.jar:1.7.0_09-icedtea] > at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_09-icedtea] > >2012-11-08 08:29:55,813 ERROR [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-49) [a707348] Installation of 10.70.36.19 has failed. Failure details: SSH command failed while executing at host '10.70.36.19', refer to logs for further information. (Stage: Running first installation script on Host) >2012-11-08 08:29:55,813 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-49) [a707348] VDS message: SSH command failed while executing at host '10.70.36.19', refer to logs for further information >2012-11-08 08:29:55,813 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-49) [a707348] Script ended, result is {1} >2012-11-08 08:29:55,813 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-4-thread-49) [a707348] After Installation pool-4-thread-49 >2012-11-08 08:29:55,814 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-49) [a707348] START, SetVdsStatusVDSCommand(HostName = rhs-gp-srv15.lab.eng.blr.redhat.com, HostId = fca61124-29a6-11e2-a257-52540035ed48, status=InstallFailed, nonOperationalReason=NONE), log id: 1bfa42a3 >2012-11-08 08:29:55,817 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-49) [a707348] FINISH, SetVdsStatusVDSCommand, log id: 1bfa42a3 >2012-11-08 08:29:56,382 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-9) Setting new tasks map. The map contains now 0 tasks >2012-11-08 08:29:56,382 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-9) Cleared all tasks of pool f4c98a0b-0578-485f-90f4-743df3d05d7e. >2012-11-08 08:29:56,385 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-9) Could not find vds that is spm and non-operational. >2012-11-08 08:30:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-52) Autorecovering hosts is disabled, skipping >2012-11-08 08:30:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-52) Autorecovering storage domains is disabled, skipping >2012-11-08 08:30:19,117 INFO [org.ovirt.engine.core.bll.SetVmTicketCommand] (ajp-/127.0.0.1:8702-13) [6af7ef2b] Running command: SetVmTicketCommand internal: false. Entities affected : ID: 28f91cc7-0841-425d-a0fd-8ea1fd15c636 Type: VM >2012-11-08 08:30:19,137 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp-/127.0.0.1:8702-13) [6af7ef2b] START, SetVmTicketVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=28f91cc7-0841-425d-a0fd-8ea1fd15c636, ticket=JaojrKpMfI8D, validTime=120,m userName=admin@internal, userId=fdfc627c-d875-11e0-90f0-83df133b58cc), log id: 7f0b464d >2012-11-08 08:30:19,173 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp-/127.0.0.1:8702-13) [6af7ef2b] FINISH, SetVmTicketVDSCommand, log id: 7f0b464d >2012-11-08 08:31:14,897 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-70) VM disrep1 661d6af9-9024-4744-91b1-258e38466815 moved from Up --> RebootInProgress >2012-11-08 08:31:14,902 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (QuartzScheduler_Worker-70) START, FullListVdsCommand( HostId = 96739be4-2990-11e2-8577-52540035ed48, vds=null, vmIds=[661d6af9-9024-4744-91b1-258e38466815]), log id: 703f6165 >2012-11-08 08:31:14,926 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (QuartzScheduler_Worker-70) FINISH, FullListVdsCommand, return: [Lorg.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcStruct;@51136775, log id: 703f6165 >2012-11-08 08:32:13,203 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-37) VM disrep1 661d6af9-9024-4744-91b1-258e38466815 moved from RebootInProgress --> Up >2012-11-08 08:32:52,569 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-10) [646ae341] START, IsValidVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 2710ae1d >2012-11-08 08:32:52,570 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-10) [646ae341] FINISH, IsValidVDSCommand, return: true, log id: 2710ae1d >2012-11-08 08:32:52,581 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp-/127.0.0.1:8702-10) [646ae341] START, IsVmDuringInitiatingVDSCommand( vmId = 67bd2d56-8e0b-4328-bcfd-ea7f4758c885), log id: 6c5c79d3 >2012-11-08 08:32:52,581 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp-/127.0.0.1:8702-10) [646ae341] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 6c5c79d3 >2012-11-08 08:32:52,613 INFO [org.ovirt.engine.core.bll.RunVmOnceCommand] (ajp-/127.0.0.1:8702-10) [646ae341] Running command: RunVmOnceCommand internal: false. Entities affected : ID: 67bd2d56-8e0b-4328-bcfd-ea7f4758c885 Type: VM, ID: 67bd2d56-8e0b-4328-bcfd-ea7f4758c885 Type: VM >2012-11-08 08:32:52,667 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsoPrefixVDSCommand] (ajp-/127.0.0.1:8702-10) [646ae341] START, IsoPrefixVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 34bdf27f >2012-11-08 08:32:52,667 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsoPrefixVDSCommand] (ajp-/127.0.0.1:8702-10) [646ae341] FINISH, IsoPrefixVDSCommand, return: /rhev/data-center/3a17dcac-a6e3-4563-a4b0-f27d89b68dd6/7a268ac7-857a-4c53-966d-247deb509161/images/11111111-1111-1111-1111-111111111111, log id: 34bdf27f >2012-11-08 08:32:52,685 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-10) [646ae341] START, CreateVmVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=67bd2d56-8e0b-4328-bcfd-ea7f4758c885, vm=org.ovirt.engine.core.common.businessentities.VM@533652a5), log id: 227b8ae6 >2012-11-08 08:32:52,697 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8702-10) [646ae341] START, CreateVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=67bd2d56-8e0b-4328-bcfd-ea7f4758c885, vm=org.ovirt.engine.core.common.businessentities.VM@533652a5), log id: 531c90dc >2012-11-08 08:32:52,760 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8702-10) [646ae341] org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand spiceSslCipherSuite=DEFAULT,memSize=512,kvmEnable=true,smp=1,emulatedMachine=rhel6.3.0,vmType=kvm,keyboardLayout=en-us,nice=0,display=qxl,smpCoresPerSocket=1,spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard,timeOffset=-43200,transparentHugePages=true,vmId=67bd2d56-8e0b-4328-bcfd-ea7f4758c885,devices=[Ljava.util.Map;@2624268,acpiEnable=true,vmName=b1,cpuType=Nehalem,custom={device_9d3a4e4b-cf63-4b40-8638-bf49f4b15d0c=VmDevice {vmId=67bd2d56-8e0b-4328-bcfd-ea7f4758c885, deviceId=9d3a4e4b-cf63-4b40-8638-bf49f4b15d0c, device=memballoon, type=balloon, bootOrder=0, specParams={model=virtio}, address={bus=0x00, domain=0x0000, type=pci, slot=0x06, function=0x0}, managed=false, plugged=true, readOnly=false, alias=balloon0}, device_9d3a4e4b-cf63-4b40-8638-bf49f4b15d0cdevice_6889cb7d-de56-47b3-a554-9f64978cb096device_c6615b71-ea57-4824-a3e7-2831e10f6e1ddevice_d81ccf23-214b-4eea-9b60-c486cf4d8b89=VmDevice {vmId=67bd2d56-8e0b-4328-bcfd-ea7f4758c885, deviceId=d81ccf23-214b-4eea-9b60-c486cf4d8b89, device=virtio-serial, type=controller, bootOrder=0, specParams={}, address={bus=0x00, domain=0x0000, type=pci, slot=0x04, function=0x0}, managed=false, plugged=true, readOnly=false, alias=virtio-serial0}, device_9d3a4e4b-cf63-4b40-8638-bf49f4b15d0cdevice_6889cb7d-de56-47b3-a554-9f64978cb096device_c6615b71-ea57-4824-a3e7-2831e10f6e1ddevice_d81ccf23-214b-4eea-9b60-c486cf4d8b89device_77fe9609-4d33-42c6-a937-4653ee6638aedevice_b4723a83-6197-433a-860b-4c19656813f8device_9d94b252-dfe9-4171-a8fe-07d467d60e34=VmDevice {vmId=67bd2d56-8e0b-4328-bcfd-ea7f4758c885, deviceId=9d94b252-dfe9-4171-a8fe-07d467d60e34, device=spicevmc, type=channel, bootOrder=0, specParams={}, address={port=3, bus=0, controller=0, type=virtio-serial}, managed=false, plugged=true, readOnly=false, alias=channel2}, device_9d3a4e4b-cf63-4b40-8638-bf49f4b15d0cdevice_6889cb7d-de56-47b3-a554-9f64978cb096device_c6615b71-ea57-4824-a3e7-2831e10f6e1ddevice_d81ccf23-214b-4eea-9b60-c486cf4d8b89device_77fe9609-4d33-42c6-a937-4653ee6638ae=VmDevice {vmId=67bd2d56-8e0b-4328-bcfd-ea7f4758c885, deviceId=77fe9609-4d33-42c6-a937-4653ee6638ae, device=unix, type=channel, bootOrder=0, specParams={}, address={port=1, bus=0, controller=0, type=virtio-serial}, managed=false, plugged=true, readOnly=false, alias=channel0}, device_9d3a4e4b-cf63-4b40-8638-bf49f4b15d0cdevice_6889cb7d-de56-47b3-a554-9f64978cb096device_c6615b71-ea57-4824-a3e7-2831e10f6e1d=VmDevice {vmId=67bd2d56-8e0b-4328-bcfd-ea7f4758c885, deviceId=c6615b71-ea57-4824-a3e7-2831e10f6e1d, device=ide, type=controller, bootOrder=0, specParams={}, address={bus=0x00, domain=0x0000, type=pci, slot=0x01, function=0x1}, managed=false, plugged=true, readOnly=false, alias=ide0}, device_9d3a4e4b-cf63-4b40-8638-bf49f4b15d0cdevice_6889cb7d-de56-47b3-a554-9f64978cb096device_c6615b71-ea57-4824-a3e7-2831e10f6e1ddevice_d81ccf23-214b-4eea-9b60-c486cf4d8b89device_77fe9609-4d33-42c6-a937-4653ee6638aedevice_b4723a83-6197-433a-860b-4c19656813f8=VmDevice {vmId=67bd2d56-8e0b-4328-bcfd-ea7f4758c885, deviceId=b4723a83-6197-433a-860b-4c19656813f8, device=unix, type=channel, bootOrder=0, specParams={}, address={port=2, bus=0, controller=0, type=virtio-serial}, managed=false, plugged=true, readOnly=false, alias=channel1}, device_9d3a4e4b-cf63-4b40-8638-bf49f4b15d0cdevice_6889cb7d-de56-47b3-a554-9f64978cb096=VmDevice {vmId=67bd2d56-8e0b-4328-bcfd-ea7f4758c885, deviceId=6889cb7d-de56-47b3-a554-9f64978cb096, device=cdrom, type=disk, bootOrder=0, specParams={}, address={unit=0, bus=1, target=0, controller=0, type=drive}, managed=false, plugged=true, readOnly=false, alias=ide0-1-0}, viodiskcache=none} >2012-11-08 08:32:52,760 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8702-10) [646ae341] FINISH, CreateVDSCommand, log id: 531c90dc >2012-11-08 08:32:52,760 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-10) [646ae341] IncreasePendingVms::CreateVmIncreasing vds rhs-gp-srv11.lab.eng.blr.redhat.com pending vcpu count, now 1. Vm: b1 >2012-11-08 08:32:52,763 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-10) [646ae341] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 227b8ae6 >2012-11-08 08:32:54,477 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-61) VM b1 67bd2d56-8e0b-4328-bcfd-ea7f4758c885 moved from WaitForLaunch --> PoweringUp >2012-11-08 08:32:54,477 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (QuartzScheduler_Worker-61) START, FullListVdsCommand( HostId = 96739be4-2990-11e2-8577-52540035ed48, vds=null, vmIds=[67bd2d56-8e0b-4328-bcfd-ea7f4758c885]), log id: 5f9bd597 >2012-11-08 08:32:54,495 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (QuartzScheduler_Worker-61) FINISH, FullListVdsCommand, return: [Lorg.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcStruct;@6fe3dd8a, log id: 5f9bd597 >2012-11-08 08:32:57,691 INFO [org.ovirt.engine.core.bll.ShutdownVmCommand] (pool-4-thread-49) [24b2da3f] Running command: ShutdownVmCommand internal: false. Entities affected : ID: 3b031d10-a679-4211-88d6-6b31c6dd7537 Type: VM >2012-11-08 08:32:57,694 INFO [org.ovirt.engine.core.bll.ShutdownVmCommand] (pool-4-thread-49) [24b2da3f] VmHandler.cs:ShutdownVmCommand:Perform: entered (Desktop FirstVM). >2012-11-08 08:32:57,694 INFO [org.ovirt.engine.core.bll.ShutdownVmCommand] (pool-4-thread-49) [24b2da3f] VmHandler.cs:ShutdownVmCommand:Perform: sending shutdown command for Desktop FirstVM. >2012-11-08 08:32:57,696 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (pool-4-thread-49) [24b2da3f] START, DestroyVmVDSCommand(HostName = rhs-arch-srv2.lab.eng.blr.redhat.com, HostId = cf017816-29a2-11e2-949e-52540035ed48, vmId=3b031d10-a679-4211-88d6-6b31c6dd7537, force=false, secondsToWait=30, gracefully=true), log id: 9dbc3f7 >2012-11-08 08:32:57,698 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (pool-4-thread-49) [24b2da3f] START, DestroyVDSCommand(HostName = rhs-arch-srv2.lab.eng.blr.redhat.com, HostId = cf017816-29a2-11e2-949e-52540035ed48, vmId=3b031d10-a679-4211-88d6-6b31c6dd7537, force=false, secondsToWait=30, gracefully=true), log id: 56e2b276 >2012-11-08 08:33:02,059 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (pool-4-thread-49) [24b2da3f] FINISH, DestroyVDSCommand, log id: 56e2b276 >2012-11-08 08:33:02,066 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (pool-4-thread-49) [24b2da3f] FINISH, DestroyVmVDSCommand, return: PoweringDown, log id: 9dbc3f7 >2012-11-08 08:33:05,063 INFO [org.ovirt.engine.core.bll.SetVmTicketCommand] (ajp-/127.0.0.1:8702-13) [2fc057fc] Running command: SetVmTicketCommand internal: false. Entities affected : ID: 68655de5-7bd9-4bdc-8628-3b2069ec312f Type: VM >2012-11-08 08:33:05,084 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp-/127.0.0.1:8702-13) [2fc057fc] START, SetVmTicketVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=68655de5-7bd9-4bdc-8628-3b2069ec312f, ticket=oUAZxS6Z8H1m, validTime=120,m userName=admin@internal, userId=fdfc627c-d875-11e0-90f0-83df133b58cc), log id: 7bb88439 >2012-11-08 08:33:05,119 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp-/127.0.0.1:8702-13) [2fc057fc] FINISH, SetVmTicketVDSCommand, log id: 7bb88439 >2012-11-08 08:33:08,728 INFO [org.ovirt.engine.core.bll.ShutdownVmCommand] (pool-4-thread-49) [178daf49] Running command: ShutdownVmCommand internal: false. Entities affected : ID: 3b031d10-a679-4211-88d6-6b31c6dd7537 Type: VM >2012-11-08 08:33:08,730 INFO [org.ovirt.engine.core.bll.ShutdownVmCommand] (pool-4-thread-49) [178daf49] VmHandler.cs:ShutdownVmCommand:Perform: entered (Desktop FirstVM). >2012-11-08 08:33:08,730 INFO [org.ovirt.engine.core.bll.ShutdownVmCommand] (pool-4-thread-49) [178daf49] VmHandler.cs:ShutdownVmCommand:Perform: Cannot shutdown Desktop FirstVM, status is not up. Stopping instead. >2012-11-08 08:33:08,735 INFO [org.ovirt.engine.core.bll.StopVmCommand] (pool-4-thread-49) [5f48d7f3] Running command: StopVmCommand internal: true. Entities affected : ID: 3b031d10-a679-4211-88d6-6b31c6dd7537 Type: VM >2012-11-08 08:33:08,737 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (pool-4-thread-49) [5f48d7f3] START, DestroyVmVDSCommand(HostName = rhs-arch-srv2.lab.eng.blr.redhat.com, HostId = cf017816-29a2-11e2-949e-52540035ed48, vmId=3b031d10-a679-4211-88d6-6b31c6dd7537, force=false, secondsToWait=0, gracefully=false), log id: 590f2db7 >2012-11-08 08:33:08,743 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (pool-4-thread-49) [5f48d7f3] START, DestroyVDSCommand(HostName = rhs-arch-srv2.lab.eng.blr.redhat.com, HostId = cf017816-29a2-11e2-949e-52540035ed48, vmId=3b031d10-a679-4211-88d6-6b31c6dd7537, force=false, secondsToWait=0, gracefully=false), log id: 2665cbc9 >2012-11-08 08:33:09,761 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (pool-4-thread-49) [5f48d7f3] FINISH, DestroyVDSCommand, log id: 2665cbc9 >2012-11-08 08:33:09,771 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (pool-4-thread-49) [5f48d7f3] FINISH, DestroyVmVDSCommand, return: Down, log id: 590f2db7 >2012-11-08 08:33:39,215 INFO [org.ovirt.engine.core.bll.SetVmTicketCommand] (ajp-/127.0.0.1:8702-5) [21f79c6] Running command: SetVmTicketCommand internal: false. Entities affected : ID: 68655de5-7bd9-4bdc-8628-3b2069ec312f Type: VM >2012-11-08 08:33:39,239 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp-/127.0.0.1:8702-5) [21f79c6] START, SetVmTicketVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=68655de5-7bd9-4bdc-8628-3b2069ec312f, ticket=vFD/hfs8Fp+0, validTime=120,m userName=admin@internal, userId=fdfc627c-d875-11e0-90f0-83df133b58cc), log id: 147096ae >2012-11-08 08:33:39,270 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp-/127.0.0.1:8702-5) [21f79c6] FINISH, SetVmTicketVDSCommand, log id: 147096ae >2012-11-08 08:33:52,913 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-29) VM b1 67bd2d56-8e0b-4328-bcfd-ea7f4758c885 moved from PoweringUp --> Up >2012-11-08 08:33:52,954 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsDynamicDataVDSCommand] (QuartzScheduler_Worker-29) START, UpdateVdsDynamicDataVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vdsDynamic=org.ovirt.engine.core.common.businessentities.VdsDynamic@3baad5fa), log id: 4e3ded41 >2012-11-08 08:33:52,956 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsDynamicDataVDSCommand] (QuartzScheduler_Worker-29) FINISH, UpdateVdsDynamicDataVDSCommand, log id: 4e3ded41 >2012-11-08 08:33:58,963 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsoPrefixVDSCommand] (ajp-/127.0.0.1:8702-15) [10e6d90c] START, IsoPrefixVDSCommand( storagePoolId = f4c98a0b-0578-485f-90f4-743df3d05d7e, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 5533582d >2012-11-08 08:33:58,963 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsoPrefixVDSCommand] (ajp-/127.0.0.1:8702-15) [10e6d90c] FINISH, IsoPrefixVDSCommand, return: /rhev/data-center/f4c98a0b-0578-485f-90f4-743df3d05d7e/7a268ac7-857a-4c53-966d-247deb509161/images/11111111-1111-1111-1111-111111111111, log id: 5533582d >2012-11-08 08:33:58,987 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-15) [10e6d90c] START, IsValidVDSCommand( storagePoolId = f4c98a0b-0578-485f-90f4-743df3d05d7e, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 20081a15 >2012-11-08 08:33:58,991 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-15) [10e6d90c] FINISH, IsValidVDSCommand, return: true, log id: 20081a15 >2012-11-08 08:33:59,006 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp-/127.0.0.1:8702-15) [10e6d90c] START, IsVmDuringInitiatingVDSCommand( vmId = 3b031d10-a679-4211-88d6-6b31c6dd7537), log id: 49979565 >2012-11-08 08:33:59,006 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp-/127.0.0.1:8702-15) [10e6d90c] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 49979565 >2012-11-08 08:33:59,044 INFO [org.ovirt.engine.core.bll.RunVmOnceCommand] (ajp-/127.0.0.1:8702-15) [10e6d90c] Running command: RunVmOnceCommand internal: false. Entities affected : ID: 3b031d10-a679-4211-88d6-6b31c6dd7537 Type: VM >2012-11-08 08:33:59,083 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-15) [10e6d90c] START, CreateVmVDSCommand(HostName = rhs-arch-srv2.lab.eng.blr.redhat.com, HostId = cf017816-29a2-11e2-949e-52540035ed48, vmId=3b031d10-a679-4211-88d6-6b31c6dd7537, vm=org.ovirt.engine.core.common.businessentities.VM@44bf7fb4), log id: 887e129 >2012-11-08 08:33:59,087 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8702-15) [10e6d90c] START, CreateVDSCommand(HostName = rhs-arch-srv2.lab.eng.blr.redhat.com, HostId = cf017816-29a2-11e2-949e-52540035ed48, vmId=3b031d10-a679-4211-88d6-6b31c6dd7537, vm=org.ovirt.engine.core.common.businessentities.VM@44bf7fb4), log id: 672a5988 >2012-11-08 08:33:59,143 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8702-15) [10e6d90c] org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand spiceSslCipherSuite=DEFAULT,memSize=4200,kvmEnable=true,smp=2,emulatedMachine=rhel6.3.0,vmType=kvm,keyboardLayout=en-us,pitReinjection=false,nice=0,display=qxl,smpCoresPerSocket=1,spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard,timeOffset=-43200,transparentHugePages=true,vmId=3b031d10-a679-4211-88d6-6b31c6dd7537,devices=[Ljava.util.Map;@144ee702,acpiEnable=true,vmName=FirstVM,cpuType=Nehalem,custom={} >2012-11-08 08:33:59,143 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8702-15) [10e6d90c] FINISH, CreateVDSCommand, log id: 672a5988 >2012-11-08 08:33:59,144 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-15) [10e6d90c] IncreasePendingVms::CreateVmIncreasing vds rhs-arch-srv2.lab.eng.blr.redhat.com pending vcpu count, now 2. Vm: FirstVM >2012-11-08 08:33:59,145 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-15) [10e6d90c] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 887e129 >2012-11-08 08:34:01,194 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-46) VM disrep2 8a0d2e26-440c-4d6c-91d3-1fc4d9b75990 moved from Up --> RebootInProgress >2012-11-08 08:34:01,198 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (QuartzScheduler_Worker-46) START, FullListVdsCommand( HostId = 96739be4-2990-11e2-8577-52540035ed48, vds=null, vmIds=[8a0d2e26-440c-4d6c-91d3-1fc4d9b75990]), log id: d1461dd >2012-11-08 08:34:01,220 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (QuartzScheduler_Worker-46) FINISH, FullListVdsCommand, return: [Lorg.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcStruct;@2c9395c5, log id: d1461dd >2012-11-08 08:34:01,237 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-59) VM FirstVM 3b031d10-a679-4211-88d6-6b31c6dd7537 moved from WaitForLaunch --> PoweringUp >2012-11-08 08:34:09,689 INFO [org.ovirt.engine.core.bll.SetVmTicketCommand] (ajp-/127.0.0.1:8702-11) [36bb9601] Running command: SetVmTicketCommand internal: false. Entities affected : ID: 3b031d10-a679-4211-88d6-6b31c6dd7537 Type: VM >2012-11-08 08:34:09,692 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp-/127.0.0.1:8702-11) [36bb9601] START, SetVmTicketVDSCommand(HostName = rhs-arch-srv2.lab.eng.blr.redhat.com, HostId = cf017816-29a2-11e2-949e-52540035ed48, vmId=3b031d10-a679-4211-88d6-6b31c6dd7537, ticket=B+ykQTm0wt28, validTime=120,m userName=admin@internal, userId=fdfc627c-d875-11e0-90f0-83df133b58cc), log id: 767d735e >2012-11-08 08:34:09,720 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp-/127.0.0.1:8702-11) [36bb9601] FINISH, SetVmTicketVDSCommand, log id: 767d735e >2012-11-08 08:34:10,865 INFO [org.ovirt.engine.core.bll.SetVmTicketCommand] (ajp-/127.0.0.1:8702-2) [39a1bb0e] Running command: SetVmTicketCommand internal: false. Entities affected : ID: 68655de5-7bd9-4bdc-8628-3b2069ec312f Type: VM >2012-11-08 08:34:10,893 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp-/127.0.0.1:8702-2) [39a1bb0e] START, SetVmTicketVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=68655de5-7bd9-4bdc-8628-3b2069ec312f, ticket=lidZ7J8MwUvk, validTime=120,m userName=admin@internal, userId=fdfc627c-d875-11e0-90f0-83df133b58cc), log id: 6c6d3b98 >2012-11-08 08:34:10,943 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp-/127.0.0.1:8702-2) [39a1bb0e] FINISH, SetVmTicketVDSCommand, log id: 6c6d3b98 >2012-11-08 08:34:55,377 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-7) VM disrep2 8a0d2e26-440c-4d6c-91d3-1fc4d9b75990 moved from RebootInProgress --> Up >2012-11-08 08:35:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-30) Autorecovering hosts is disabled, skipping >2012-11-08 08:35:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-30) Autorecovering storage domains is disabled, skipping >2012-11-08 08:35:00,922 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-31) [6143973] VM FirstVM 3b031d10-a679-4211-88d6-6b31c6dd7537 moved from PoweringUp --> Up >2012-11-08 08:35:00,928 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsDynamicDataVDSCommand] (QuartzScheduler_Worker-31) [6143973] START, UpdateVdsDynamicDataVDSCommand(HostName = rhs-arch-srv2.lab.eng.blr.redhat.com, HostId = cf017816-29a2-11e2-949e-52540035ed48, vdsDynamic=org.ovirt.engine.core.common.businessentities.VdsDynamic@265dc965), log id: 714896c1 >2012-11-08 08:35:00,930 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsDynamicDataVDSCommand] (QuartzScheduler_Worker-31) [6143973] FINISH, UpdateVdsDynamicDataVDSCommand, log id: 714896c1 >2012-11-08 08:35:32,594 INFO [org.ovirt.engine.core.bll.SetVmTicketCommand] (ajp-/127.0.0.1:8702-7) [7fe16b7] Running command: SetVmTicketCommand internal: false. Entities affected : ID: 68655de5-7bd9-4bdc-8628-3b2069ec312f Type: VM >2012-11-08 08:35:32,617 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp-/127.0.0.1:8702-7) [7fe16b7] START, SetVmTicketVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=68655de5-7bd9-4bdc-8628-3b2069ec312f, ticket=O6I8OeB8yzJT, validTime=120,m userName=admin@internal, userId=fdfc627c-d875-11e0-90f0-83df133b58cc), log id: 20571d1 >2012-11-08 08:35:32,646 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp-/127.0.0.1:8702-7) [7fe16b7] FINISH, SetVmTicketVDSCommand, log id: 20571d1 >2012-11-08 08:37:27,752 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-46) VM b4 e228f541-76b7-4a9b-b313-a6bacc2b73b9 moved from Up --> RebootInProgress >2012-11-08 08:37:27,753 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-46) VM disrep4 28f91cc7-0841-425d-a0fd-8ea1fd15c636 moved from Up --> RebootInProgress >2012-11-08 08:37:27,757 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (QuartzScheduler_Worker-46) START, FullListVdsCommand( HostId = 96739be4-2990-11e2-8577-52540035ed48, vds=null, vmIds=[e228f541-76b7-4a9b-b313-a6bacc2b73b9, 28f91cc7-0841-425d-a0fd-8ea1fd15c636]), log id: 5fabfdfd >2012-11-08 08:37:27,791 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (QuartzScheduler_Worker-46) FINISH, FullListVdsCommand, return: [Lorg.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcStruct;@14b01e1d, log id: 5fabfdfd >2012-11-08 08:37:35,418 INFO [org.ovirt.engine.core.bll.SetVmTicketCommand] (ajp-/127.0.0.1:8702-1) [62231a44] Running command: SetVmTicketCommand internal: false. Entities affected : ID: 68655de5-7bd9-4bdc-8628-3b2069ec312f Type: VM >2012-11-08 08:37:35,440 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp-/127.0.0.1:8702-1) [62231a44] START, SetVmTicketVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=68655de5-7bd9-4bdc-8628-3b2069ec312f, ticket=pQuRbfsVZhNf, validTime=120,m userName=admin@internal, userId=fdfc627c-d875-11e0-90f0-83df133b58cc), log id: 16c40703 >2012-11-08 08:37:35,469 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp-/127.0.0.1:8702-1) [62231a44] FINISH, SetVmTicketVDSCommand, log id: 16c40703 >2012-11-08 08:38:08,454 INFO [org.ovirt.engine.core.bll.SetVmTicketCommand] (ajp-/127.0.0.1:8702-5) [41b9237e] Running command: SetVmTicketCommand internal: false. Entities affected : ID: 68655de5-7bd9-4bdc-8628-3b2069ec312f Type: VM >2012-11-08 08:38:08,487 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp-/127.0.0.1:8702-5) [41b9237e] START, SetVmTicketVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=68655de5-7bd9-4bdc-8628-3b2069ec312f, ticket=sZALS7pQl7lK, validTime=120,m userName=admin@internal, userId=fdfc627c-d875-11e0-90f0-83df133b58cc), log id: 7169df75 >2012-11-08 08:38:08,518 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp-/127.0.0.1:8702-5) [41b9237e] FINISH, SetVmTicketVDSCommand, log id: 7169df75 >2012-11-08 08:38:20,471 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-4) VM b4 e228f541-76b7-4a9b-b313-a6bacc2b73b9 moved from RebootInProgress --> Up >2012-11-08 08:38:24,659 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-21) VM disrep4 28f91cc7-0841-425d-a0fd-8ea1fd15c636 moved from RebootInProgress --> Up >2012-11-08 08:39:52,949 INFO [org.ovirt.engine.core.bll.LogoutUserCommand] (ajp-/127.0.0.1:8702-5) [e630007] Running command: LogoutUserCommand internal: false. >2012-11-08 08:40:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-2) Autorecovering hosts is disabled, skipping >2012-11-08 08:40:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-2) Autorecovering storage domains is disabled, skipping >2012-11-08 08:40:00,344 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-12) VM FirstVM 3b031d10-a679-4211-88d6-6b31c6dd7537 moved from Up --> RebootInProgress >2012-11-08 08:40:00,346 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (QuartzScheduler_Worker-12) START, FullListVdsCommand( HostId = cf017816-29a2-11e2-949e-52540035ed48, vds=null, vmIds=[3b031d10-a679-4211-88d6-6b31c6dd7537]), log id: 5ac9b04b >2012-11-08 08:40:00,369 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (QuartzScheduler_Worker-12) FINISH, FullListVdsCommand, return: [Lorg.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcStruct;@6f553a7d, log id: 5ac9b04b >2012-11-08 08:40:34,264 INFO [org.ovirt.engine.core.bll.LoginAdminUserCommand] (ajp-/127.0.0.1:8702-11) Checking if user admin@internal is an admin, result true >2012-11-08 08:40:34,264 INFO [org.ovirt.engine.core.bll.LoginAdminUserCommand] (ajp-/127.0.0.1:8702-11) Running command: LoginAdminUserCommand internal: false. >2012-11-08 08:40:53,820 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-68) VM FirstVM 3b031d10-a679-4211-88d6-6b31c6dd7537 moved from RebootInProgress --> Up >2012-11-08 08:42:29,760 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (ajp-/127.0.0.1:8702-10) Lock Acquired to object EngineLock [exclusiveLocks= key: fca61124-29a6-11e2-a257-52540035ed48 value: VDS >, sharedLocks= ] >2012-11-08 08:42:29,780 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (pool-4-thread-10) [1e7b643c] Running command: ActivateVdsCommand internal: false. Entities affected : ID: fca61124-29a6-11e2-a257-52540035ed48 Type: VDS >2012-11-08 08:42:29,784 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-10) [1e7b643c] START, SetVdsStatusVDSCommand(HostName = rhs-gp-srv15.lab.eng.blr.redhat.com, HostId = fca61124-29a6-11e2-a257-52540035ed48, status=Unassigned, nonOperationalReason=NONE), log id: 670b78a >2012-11-08 08:42:29,786 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-10) [1e7b643c] FINISH, SetVdsStatusVDSCommand, log id: 670b78a >2012-11-08 08:42:29,789 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (pool-4-thread-10) [1e7b643c] START, ActivateVdsVDSCommand(HostName = rhs-gp-srv15.lab.eng.blr.redhat.com, HostId = fca61124-29a6-11e2-a257-52540035ed48), log id: b4ba980 >2012-11-08 08:42:29,795 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (pool-4-thread-10) [1e7b643c] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 08:42:29,799 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (pool-4-thread-10) [1e7b643c] ResourceManager::activateVds - failed to get VDS = fca61124-29a6-11e2-a257-52540035ed48 capabilities with error: VDSNetworkException: . >2012-11-08 08:42:29,799 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (pool-4-thread-10) [1e7b643c] ResourceManager::activateVds - failed to activate VDS = fca61124-29a6-11e2-a257-52540035ed48 >2012-11-08 08:42:29,803 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (pool-4-thread-10) [12c1828] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: fca61124-29a6-11e2-a257-52540035ed48 Type: VDS >2012-11-08 08:42:29,807 WARN [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (pool-4-thread-10) [1d5c6306] CanDoAction of action HandleVdsVersion failed. Reasons:VDS_CANNOT_CHECK_VERSION_HOST_NON_RESPONSIVE >2012-11-08 08:42:29,807 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (pool-4-thread-10) [1d5c6306] FINISH, ActivateVdsVDSCommand, log id: b4ba980 >2012-11-08 08:42:29,832 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (pool-4-thread-10) [1d5c6306] Lock freed to object EngineLock [exclusiveLocks= key: fca61124-29a6-11e2-a257-52540035ed48 value: VDS >, sharedLocks= ] >2012-11-08 08:42:30,766 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-51) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 08:42:30,766 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-51) ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = fca61124-29a6-11e2-a257-52540035ed48 : rhs-gp-srv15.lab.eng.blr.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-11-08 08:42:32,775 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-60) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 08:42:32,775 ERROR [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-60) VDS::handleNetworkException Server failed to respond, vds_id = fca61124-29a6-11e2-a257-52540035ed48, vds_name = rhs-gp-srv15.lab.eng.blr.redhat.com, error = VDSNetworkException: >2012-11-08 08:42:32,783 INFO [org.ovirt.engine.core.bll.VdsEventListener] (pool-4-thread-10) ResourceManager::vdsNotResponding entered for Host fca61124-29a6-11e2-a257-52540035ed48, 10.70.36.19 >2012-11-08 08:42:32,793 WARN [org.ovirt.engine.core.bll.VdsNotRespondingTreatmentCommand] (pool-4-thread-10) [3ac51047] CanDoAction of action VdsNotRespondingTreatment failed. Reasons:VDS_FENCING_DISABLED >2012-11-08 08:42:34,797 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-56) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 08:42:36,807 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-66) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 08:42:38,819 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-70) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 08:42:40,846 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-73) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 08:42:42,859 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-90) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 08:42:44,870 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-87) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 08:42:46,882 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-62) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 08:42:48,894 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-5) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 08:42:50,919 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-6) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 08:42:52,929 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-24) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 08:42:54,939 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-11) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 08:42:56,947 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-22) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 08:42:58,959 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-26) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 08:43:00,863 INFO [org.ovirt.engine.core.bll.MaintananceNumberOfVdssCommand] (pool-4-thread-45) [7a4988db] Running command: MaintananceNumberOfVdssCommand internal: false. Entities affected : ID: fca61124-29a6-11e2-a257-52540035ed48 Type: VDS >2012-11-08 08:43:00,875 INFO [org.ovirt.engine.core.bll.MaintananceVdsCommand] (pool-4-thread-45) [7a4988db] Running command: MaintananceVdsCommand internal: true. Entities affected : ID: fca61124-29a6-11e2-a257-52540035ed48 Type: VDS >2012-11-08 08:43:00,877 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-45) [7a4988db] START, SetVdsStatusVDSCommand(HostName = rhs-gp-srv15.lab.eng.blr.redhat.com, HostId = fca61124-29a6-11e2-a257-52540035ed48, status=Maintenance, nonOperationalReason=NONE), log id: 6fa05040 >2012-11-08 08:43:00,880 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-45) [7a4988db] FINISH, SetVdsStatusVDSCommand, log id: 6fa05040 >2012-11-08 08:43:08,748 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (ajp-/127.0.0.1:8702-6) Lock Acquired to object EngineLock [exclusiveLocks= key: f047378c-29a6-11e2-8a79-52540035ed48 value: VDS >, sharedLocks= ] >2012-11-08 08:43:08,767 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (pool-4-thread-45) [514d7782] Running command: ActivateVdsCommand internal: false. Entities affected : ID: f047378c-29a6-11e2-8a79-52540035ed48 Type: VDS >2012-11-08 08:43:08,771 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-45) [514d7782] START, SetVdsStatusVDSCommand(HostName = rhs-gp-srv12.lab.eng.blr.redhat.com, HostId = f047378c-29a6-11e2-8a79-52540035ed48, status=Unassigned, nonOperationalReason=NONE), log id: 3758381f >2012-11-08 08:43:08,774 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-45) [514d7782] FINISH, SetVdsStatusVDSCommand, log id: 3758381f >2012-11-08 08:43:08,776 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (pool-4-thread-45) [514d7782] START, ActivateVdsVDSCommand(HostName = rhs-gp-srv12.lab.eng.blr.redhat.com, HostId = f047378c-29a6-11e2-8a79-52540035ed48), log id: 5f712b8b >2012-11-08 08:43:08,782 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (pool-4-thread-45) [514d7782] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 08:43:08,785 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (pool-4-thread-45) [514d7782] ResourceManager::activateVds - failed to get VDS = f047378c-29a6-11e2-8a79-52540035ed48 capabilities with error: VDSNetworkException: . >2012-11-08 08:43:08,785 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (pool-4-thread-45) [514d7782] ResourceManager::activateVds - failed to activate VDS = f047378c-29a6-11e2-8a79-52540035ed48 >2012-11-08 08:43:08,789 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (pool-4-thread-45) [5aafd613] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: f047378c-29a6-11e2-8a79-52540035ed48 Type: VDS >2012-11-08 08:43:08,793 WARN [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (pool-4-thread-45) [542b5d13] CanDoAction of action HandleVdsVersion failed. Reasons:VDS_CANNOT_CHECK_VERSION_HOST_NON_RESPONSIVE >2012-11-08 08:43:08,793 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (pool-4-thread-45) [542b5d13] FINISH, ActivateVdsVDSCommand, log id: 5f712b8b >2012-11-08 08:43:08,816 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (pool-4-thread-45) [542b5d13] Lock freed to object EngineLock [exclusiveLocks= key: f047378c-29a6-11e2-8a79-52540035ed48 value: VDS >, sharedLocks= ] >2012-11-08 08:43:10,039 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-72) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 08:43:10,039 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-72) ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = f047378c-29a6-11e2-8a79-52540035ed48 : rhs-gp-srv12.lab.eng.blr.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-11-08 08:43:12,046 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-70) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 08:43:12,047 ERROR [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-70) VDS::handleNetworkException Server failed to respond, vds_id = f047378c-29a6-11e2-8a79-52540035ed48, vds_name = rhs-gp-srv12.lab.eng.blr.redhat.com, error = VDSNetworkException: >2012-11-08 08:43:12,056 INFO [org.ovirt.engine.core.bll.VdsEventListener] (pool-4-thread-45) ResourceManager::vdsNotResponding entered for Host f047378c-29a6-11e2-8a79-52540035ed48, 10.70.36.16 >2012-11-08 08:43:12,068 WARN [org.ovirt.engine.core.bll.VdsNotRespondingTreatmentCommand] (pool-4-thread-45) [569dd848] CanDoAction of action VdsNotRespondingTreatment failed. Reasons:VDS_FENCING_DISABLED >2012-11-08 08:43:14,067 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-85) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 08:43:16,079 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-83) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 08:43:18,091 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-69) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 08:43:20,102 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-84) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 08:43:22,114 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-5) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 08:43:24,126 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-3) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 08:43:26,138 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-10) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 08:43:26,776 INFO [org.ovirt.engine.core.bll.MaintananceNumberOfVdssCommand] (pool-4-thread-45) [11e865f7] Running command: MaintananceNumberOfVdssCommand internal: false. Entities affected : ID: f047378c-29a6-11e2-8a79-52540035ed48 Type: VDS >2012-11-08 08:43:26,791 INFO [org.ovirt.engine.core.bll.MaintananceVdsCommand] (pool-4-thread-45) [11e865f7] Running command: MaintananceVdsCommand internal: true. Entities affected : ID: f047378c-29a6-11e2-8a79-52540035ed48 Type: VDS >2012-11-08 08:43:26,795 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-45) [11e865f7] START, SetVdsStatusVDSCommand(HostName = rhs-gp-srv12.lab.eng.blr.redhat.com, HostId = f047378c-29a6-11e2-8a79-52540035ed48, status=Maintenance, nonOperationalReason=NONE), log id: 7c3692bd >2012-11-08 08:43:26,797 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-45) [11e865f7] FINISH, SetVdsStatusVDSCommand, log id: 7c3692bd >2012-11-08 08:43:29,765 INFO [org.ovirt.engine.core.bll.RemoveVdsCommand] (ajp-/127.0.0.1:8702-15) Lock Acquired to object EngineLock [exclusiveLocks= key: fca61124-29a6-11e2-a257-52540035ed48 value: VDS >, sharedLocks= ] >2012-11-08 08:43:29,770 INFO [org.ovirt.engine.core.bll.RemoveVdsCommand] (pool-4-thread-45) [548d0ae1] Running command: RemoveVdsCommand internal: false. Entities affected : ID: fca61124-29a6-11e2-a257-52540035ed48 Type: VDS >2012-11-08 08:43:29,777 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (pool-4-thread-45) [548d0ae1] START, RemoveVdsVDSCommand( HostId = fca61124-29a6-11e2-a257-52540035ed48), log id: 3a646297 >2012-11-08 08:43:29,777 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (pool-4-thread-45) [548d0ae1] vdsManager::disposing >2012-11-08 08:43:29,777 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (pool-4-thread-45) [548d0ae1] FINISH, RemoveVdsVDSCommand, log id: 3a646297 >2012-11-08 08:43:29,778 INFO [org.ovirt.engine.core.bll.RemoveVdsCommand] (pool-4-thread-45) [548d0ae1] Lock freed to object EngineLock [exclusiveLocks= key: fca61124-29a6-11e2-a257-52540035ed48 value: VDS >, sharedLocks= ] >2012-11-08 08:43:32,858 INFO [org.ovirt.engine.core.bll.RemoveVdsCommand] (ajp-/127.0.0.1:8702-11) Lock Acquired to object EngineLock [exclusiveLocks= key: f047378c-29a6-11e2-8a79-52540035ed48 value: VDS >, sharedLocks= ] >2012-11-08 08:43:32,865 INFO [org.ovirt.engine.core.bll.RemoveVdsCommand] (pool-4-thread-10) [49afc904] Running command: RemoveVdsCommand internal: false. Entities affected : ID: f047378c-29a6-11e2-8a79-52540035ed48 Type: VDS >2012-11-08 08:43:32,872 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (pool-4-thread-10) [49afc904] START, RemoveVdsVDSCommand( HostId = f047378c-29a6-11e2-8a79-52540035ed48), log id: 3506a34e >2012-11-08 08:43:32,872 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (pool-4-thread-10) [49afc904] vdsManager::disposing >2012-11-08 08:43:32,872 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (pool-4-thread-10) [49afc904] FINISH, RemoveVdsVDSCommand, log id: 3506a34e >2012-11-08 08:43:32,873 INFO [org.ovirt.engine.core.bll.RemoveVdsCommand] (pool-4-thread-10) [49afc904] Lock freed to object EngineLock [exclusiveLocks= key: f047378c-29a6-11e2-8a79-52540035ed48 value: VDS >, sharedLocks= ] >2012-11-08 08:44:17,728 ERROR [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (ajp-/127.0.0.1:8702-16) Could not connect to server 10.70.36.12: javax.naming.AuthenticationException: SSH authentication to '10.70.36.12:22' failed make sure host is configured for password authentication > at org.ovirt.engine.core.utils.ssh.SSHClient.authenticate(SSHClient.java:294) [engine-utils.jar:] > at org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH._doConnect(VdsInstallerSSH.java:209) [engine-utils.jar:] > at org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH.connect(VdsInstallerSSH.java:275) [engine-utils.jar:] > at org.ovirt.engine.core.bll.VdsInstallHelper.connectToServer(VdsInstallHelper.java:33) [engine-bll.jar:] > at org.ovirt.engine.core.bll.AddVdsCommand.validateHostUniqueness(AddVdsCommand.java:353) [engine-bll.jar:] > at org.ovirt.engine.core.bll.AddVdsCommand.canDoAction(AddVdsCommand.java:315) [engine-bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.internalCanDoAction(CommandBase.java:510) [engine-bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:278) [engine-bll.jar:] > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:336) [engine-bll.jar:] > at org.ovirt.engine.core.bll.Backend.RunAction(Backend.java:294) [engine-bll.jar:] > at sun.reflect.GeneratedMethodAccessor99.invoke(Unknown Source) [:1.7.0_09-icedtea] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_09-icedtea] > at java.lang.reflect.Method.invoke(Method.java:601) [rt.jar:1.7.0_09-icedtea] > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) [jboss-as-ee.jar:7.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:374) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.ovirt.engine.core.utils.ThreadLocalSessionCleanerInterceptor.injectWebContextToThreadLocal(ThreadLocalSessionCleanerInterceptor.java:11) [engine-utils.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_09-icedtea] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_09-icedtea] > at java.lang.reflect.Method.invoke(Method.java:601) [rt.jar:1.7.0_09-icedtea] > at org.jboss.as.ee.component.ManagedReferenceLifecycleMethodInterceptorFactory$ManagedReferenceLifecycleMethodInterceptor.processInvocation(ManagedReferenceLifecycleMethodInterceptorFactory.java:123) [jboss-as-ee.jar:7.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:36) [jboss-as-ee.jar:7.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) [jboss-as-ejb3.jar:7.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) [jboss-as-ee.jar:7.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:53) [jboss-as-ejb3.jar:7.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:210) [jboss-as-ejb3.jar:7.1.2.Final-redhat-1] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:362) [jboss-as-ejb3.jar:7.1.2.Final-redhat-1] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:193) [jboss-as-ejb3.jar:7.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [jboss-as-ejb3.jar:7.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:42) [jboss-as-ejb3.jar:7.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) [jboss-as-ejb3.jar:7.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) [jboss-as-ee.jar:7.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) [jboss-as-ee.jar:7.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) [jboss-as-ee.jar:7.1.2.Final-redhat-1] > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:176) [jboss-as-ee.jar:7.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) [jboss-as-ee.jar:7.1.2.Final-redhat-1] > at org.ovirt.engine.core.common.interfaces.BackendLocal$$$view9.RunAction(Unknown Source) [engine-common.jar:] > at org.ovirt.engine.ui.frontend.server.gwt.GenericApiGWTServiceImpl.RunAction(GenericApiGWTServiceImpl.java:120) > at sun.reflect.GeneratedMethodAccessor98.invoke(Unknown Source) [:1.7.0_09-icedtea] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_09-icedtea] > at java.lang.reflect.Method.invoke(Method.java:601) [rt.jar:1.7.0_09-icedtea] > at com.google.gwt.rpc.server.RPC.invokeAndStreamResponse(RPC.java:196) > at com.google.gwt.rpc.server.RpcServlet.processCall(RpcServlet.java:161) > at com.google.gwt.rpc.server.RpcServlet.processPost(RpcServlet.java:222) > at com.google.gwt.user.server.rpc.AbstractRemoteServiceServlet.doPost(AbstractRemoteServiceServlet.java:62) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:754) [jboss-servlet-api_3.0_spec.jar:1.0.1.Final-redhat-1] > at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) [jboss-servlet-api_3.0_spec.jar:1.0.1.Final-redhat-1] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:329) > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) > at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:275) > at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:161) > at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:489) > at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:153) > at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:155) > at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) > at org.jboss.web.rewrite.RewriteValve.invoke(RewriteValve.java:466) > at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) > at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:368) > at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:505) > at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:452) > at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:931) > at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_09-icedtea] > >2012-11-08 08:44:17,732 WARN [org.ovirt.engine.core.bll.AddVdsCommand] (ajp-/127.0.0.1:8702-16) [60c0ee3b] CanDoAction of action AddVds failed. Reasons:VDS_CANNOT_CONNECT_TO_SERVER,VAR__ACTION__ADD,VAR__TYPE__HOST >2012-11-08 08:44:18,030 INFO [org.ovirt.engine.core.bll.RemoveVmCommand] (ajp-/127.0.0.1:8702-4) [5350a859] Lock Acquired to object EngineLock [exclusiveLocks= key: a1059fbe-4eb3-493e-b8b6-b14ecba6fbe2 value: VM >, sharedLocks= ] >2012-11-08 08:44:18,037 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-4) [5350a859] START, IsValidVDSCommand( storagePoolId = f4c98a0b-0578-485f-90f4-743df3d05d7e, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 4f008e3c >2012-11-08 08:44:18,038 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-4) [5350a859] FINISH, IsValidVDSCommand, return: true, log id: 4f008e3c >2012-11-08 08:44:18,054 INFO [org.ovirt.engine.core.bll.RemoveVmCommand] (ajp-/127.0.0.1:8702-4) [5350a859] Running command: RemoveVmCommand internal: false. Entities affected : ID: a1059fbe-4eb3-493e-b8b6-b14ecba6fbe2 Type: VM >2012-11-08 08:44:18,055 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (ajp-/127.0.0.1:8702-4) [5350a859] START, SetVmStatusVDSCommand( vmId = a1059fbe-4eb3-493e-b8b6-b14ecba6fbe2, status = ImageLocked), log id: 88cedea >2012-11-08 08:44:18,058 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (ajp-/127.0.0.1:8702-4) [5350a859] FINISH, SetVmStatusVDSCommand, log id: 88cedea >2012-11-08 08:44:18,060 INFO [org.ovirt.engine.core.bll.RemoveVmCommand] (ajp-/127.0.0.1:8702-4) [5350a859] Lock freed to object EngineLock [exclusiveLocks= key: a1059fbe-4eb3-493e-b8b6-b14ecba6fbe2 value: VM >, sharedLocks= ] >2012-11-08 08:44:18,061 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.RemoveVMVDSCommand] (ajp-/127.0.0.1:8702-4) [5350a859] START, RemoveVMVDSCommand( storagePoolId = f4c98a0b-0578-485f-90f4-743df3d05d7e, ignoreFailoverLimit = false, compatabilityVersion = null, storageDomainId = 00000000-0000-0000-0000-000000000000, vmGuid = a1059fbe-4eb3-493e-b8b6-b14ecba6fbe2), log id: 4bd35c77 >2012-11-08 08:44:18,103 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.RemoveVMVDSCommand] (ajp-/127.0.0.1:8702-4) [5350a859] FINISH, RemoveVMVDSCommand, log id: 4bd35c77 >2012-11-08 08:44:18,109 INFO [org.ovirt.engine.core.bll.RemoveAllVmImagesCommand] (ajp-/127.0.0.1:8702-4) [215baf85] Running command: RemoveAllVmImagesCommand internal: true. Entities affected : ID: a1059fbe-4eb3-493e-b8b6-b14ecba6fbe2 Type: VM >2012-11-08 08:44:18,119 INFO [org.ovirt.engine.core.bll.RemoveImageCommand] (ajp-/127.0.0.1:8702-4) [8ca1950] Running command: RemoveImageCommand internal: true. Entities affected : ID: 00000000-0000-0000-0000-000000000000 Type: Storage >2012-11-08 08:44:18,123 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] (ajp-/127.0.0.1:8702-4) [8ca1950] START, DeleteImageGroupVDSCommand( storagePoolId = f4c98a0b-0578-485f-90f4-743df3d05d7e, ignoreFailoverLimit = false, compatabilityVersion = 3.1, storageDomainId = 00068b9a-c8e3-4e01-b2c4-963931bfae65, imageGroupId = 1d59d61e-5a79-471a-b722-4926b9866b9a, postZeros = false, forceDelete = false), log id: 72e97c55 >2012-11-08 08:44:18,355 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] (ajp-/127.0.0.1:8702-4) [8ca1950] FINISH, DeleteImageGroupVDSCommand, log id: 72e97c55 >2012-11-08 08:44:18,358 INFO [org.ovirt.engine.core.bll.EntityAsyncTask] (ajp-/127.0.0.1:8702-4) [8ca1950] EntityAsyncTask::Adding EntityMultiAsyncTasks object for entity a1059fbe-4eb3-493e-b8b6-b14ecba6fbe2 >2012-11-08 08:44:18,358 INFO [org.ovirt.engine.core.bll.EntityMultiAsyncTasks] (ajp-/127.0.0.1:8702-4) [8ca1950] EntityMultiAsyncTasks::AttachTask: Attaching task 874050fd-603c-4bfe-9827-693ca8e02a8b to entity a1059fbe-4eb3-493e-b8b6-b14ecba6fbe2. >2012-11-08 08:44:18,360 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskUtils] (ajp-/127.0.0.1:8702-4) [8ca1950] Adding task 874050fd-603c-4bfe-9827-693ca8e02a8b to DataBase >2012-11-08 08:44:18,369 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (ajp-/127.0.0.1:8702-4) [8ca1950] Adding task 874050fd-603c-4bfe-9827-693ca8e02a8b (Parent Command RemoveVm, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters), polling hasn't started yet.. >2012-11-08 08:44:18,459 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (ajp-/127.0.0.1:8702-4) [8ca1950] BaseAsyncTask::StartPollingTask: Starting to poll task 874050fd-603c-4bfe-9827-693ca8e02a8b. >2012-11-08 08:44:18,459 INFO [org.ovirt.engine.core.bll.EntityMultiAsyncTasks] (ajp-/127.0.0.1:8702-4) [8ca1950] EntityMultiAsyncTasks::StartPollingTask: Current Action Type for entity a1059fbe-4eb3-493e-b8b6-b14ecba6fbe2 is RemoveVm (determined by task 874050fd-603c-4bfe-9827-693ca8e02a8b) >2012-11-08 08:44:18,939 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-16) Polling and updating Async Tasks: 1 tasks, 1 tasks to poll now >2012-11-08 08:44:18,961 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (QuartzScheduler_Worker-16) SPMAsyncTask::PollTask: Polling task 874050fd-603c-4bfe-9827-693ca8e02a8b (Parent Command RemoveVm, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status finished, result 'success'. >2012-11-08 08:44:18,964 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (QuartzScheduler_Worker-16) BaseAsyncTask::OnTaskEndSuccess: Task 874050fd-603c-4bfe-9827-693ca8e02a8b (Parent Command RemoveVm, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended successfully. >2012-11-08 08:44:18,964 INFO [org.ovirt.engine.core.bll.EntityAsyncTask] (QuartzScheduler_Worker-16) EntityAsyncTask::EndActionIfNecessary: All tasks of entity a1059fbe-4eb3-493e-b8b6-b14ecba6fbe2 has ended -> executing EndAction >2012-11-08 08:44:18,965 INFO [org.ovirt.engine.core.bll.EntityAsyncTask] (QuartzScheduler_Worker-16) EntityAsyncTask::EndAction: Ending action for 1 tasks (entity ID: a1059fbe-4eb3-493e-b8b6-b14ecba6fbe2): calling EndAction for action type RemoveVm. >2012-11-08 08:44:18,965 INFO [org.ovirt.engine.core.bll.EntityAsyncTask] (pool-4-thread-45) EntityAsyncTask::EndCommandAction [within thread]context: Attempting to EndAction RemoveVm >2012-11-08 08:44:18,976 INFO [org.ovirt.engine.core.bll.RemoveVmCommand] (pool-4-thread-45) [5350a859] Ending command successfully: org.ovirt.engine.core.bll.RemoveVmCommand >2012-11-08 08:44:18,977 INFO [org.ovirt.engine.core.bll.RemoveVmCommand] (pool-4-thread-45) [5350a859] Lock Acquired to object EngineLock [exclusiveLocks= key: a1059fbe-4eb3-493e-b8b6-b14ecba6fbe2 value: VM >, sharedLocks= ] >2012-11-08 08:44:18,987 INFO [org.ovirt.engine.core.bll.MacPoolManager] (pool-4-thread-45) [5350a859] MacPoolManager::freeMac(mac = 00:1a:4a:46:22:59) - entered >2012-11-08 08:44:18,987 INFO [org.ovirt.engine.core.bll.MacPoolManager] (pool-4-thread-45) [5350a859] MacPoolManager::freeMac(mac = 00:1a:4a:46:22:60) - entered >2012-11-08 08:44:19,007 INFO [org.ovirt.engine.core.bll.RemoveVmCommand] (pool-4-thread-45) [5350a859] Lock freed to object EngineLock [exclusiveLocks= key: a1059fbe-4eb3-493e-b8b6-b14ecba6fbe2 value: VM >, sharedLocks= ] >2012-11-08 08:44:19,016 INFO [org.ovirt.engine.core.bll.EntityAsyncTask] (pool-4-thread-45) EntityAsyncTask::HandleEndActionResult [within thread]: EndAction for action type RemoveVm completed, handling the result. >2012-11-08 08:44:19,016 INFO [org.ovirt.engine.core.bll.EntityAsyncTask] (pool-4-thread-45) EntityAsyncTask::HandleEndActionResult [within thread]: EndAction for action type RemoveVm succeeded, clearing tasks. >2012-11-08 08:44:19,019 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (pool-4-thread-45) SPMAsyncTask::ClearAsyncTask: Attempting to clear task 874050fd-603c-4bfe-9827-693ca8e02a8b >2012-11-08 08:44:19,019 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (pool-4-thread-45) START, SPMClearTaskVDSCommand( storagePoolId = f4c98a0b-0578-485f-90f4-743df3d05d7e, ignoreFailoverLimit = false, compatabilityVersion = null, taskId = 874050fd-603c-4bfe-9827-693ca8e02a8b), log id: 7f85cc11 >2012-11-08 08:44:19,021 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (pool-4-thread-45) START, HSMClearTaskVDSCommand(HostName = rhs-arch-srv2.lab.eng.blr.redhat.com, HostId = cf017816-29a2-11e2-949e-52540035ed48, taskId=874050fd-603c-4bfe-9827-693ca8e02a8b), log id: e064693 >2012-11-08 08:44:19,058 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (pool-4-thread-45) FINISH, HSMClearTaskVDSCommand, log id: e064693 >2012-11-08 08:44:19,059 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (pool-4-thread-45) FINISH, SPMClearTaskVDSCommand, log id: 7f85cc11 >2012-11-08 08:44:19,060 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (pool-4-thread-45) BaseAsyncTask::RemoveTaskFromDB: Removed task 874050fd-603c-4bfe-9827-693ca8e02a8b from DataBase >2012-11-08 08:44:19,060 INFO [org.ovirt.engine.core.bll.EntityAsyncTask] (pool-4-thread-45) EntityAsyncTask::HandleEndActionResult [within thread]: Removing EntityMultiAsyncTasks object for entity a1059fbe-4eb3-493e-b8b6-b14ecba6fbe2 >2012-11-08 08:44:23,540 INFO [org.ovirt.engine.core.bll.SetVmTicketCommand] (ajp-/127.0.0.1:8702-10) [625cb519] Running command: SetVmTicketCommand internal: false. Entities affected : ID: 67bd2d56-8e0b-4328-bcfd-ea7f4758c885 Type: VM >2012-11-08 08:44:23,567 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp-/127.0.0.1:8702-10) [625cb519] START, SetVmTicketVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=67bd2d56-8e0b-4328-bcfd-ea7f4758c885, ticket=6VvdEQGOSZ0r, validTime=120,m userName=admin@internal, userId=fdfc627c-d875-11e0-90f0-83df133b58cc), log id: 3202549d >2012-11-08 08:44:23,601 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp-/127.0.0.1:8702-10) [625cb519] FINISH, SetVmTicketVDSCommand, log id: 3202549d >2012-11-08 08:45:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-89) Autorecovering hosts is disabled, skipping >2012-11-08 08:45:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-89) Autorecovering storage domains is disabled, skipping >2012-11-08 08:45:26,393 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-51) Setting new tasks map. The map contains now 0 tasks >2012-11-08 08:45:26,393 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-51) Cleared all tasks of pool f4c98a0b-0578-485f-90f4-743df3d05d7e. >2012-11-08 08:45:26,396 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-51) Could not find vds that is spm and non-operational. >2012-11-08 08:47:42,883 ERROR [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (ajp-/127.0.0.1:8702-9) Could not connect to server 10.70.36.12: javax.naming.AuthenticationException: SSH authentication to '10.70.36.12:22' failed make sure host is configured for password authentication > at org.ovirt.engine.core.utils.ssh.SSHClient.authenticate(SSHClient.java:294) [engine-utils.jar:] > at org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH._doConnect(VdsInstallerSSH.java:209) [engine-utils.jar:] > at org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH.connect(VdsInstallerSSH.java:275) [engine-utils.jar:] > at org.ovirt.engine.core.bll.VdsInstallHelper.connectToServer(VdsInstallHelper.java:33) [engine-bll.jar:] > at org.ovirt.engine.core.bll.AddVdsCommand.validateHostUniqueness(AddVdsCommand.java:353) [engine-bll.jar:] > at org.ovirt.engine.core.bll.AddVdsCommand.canDoAction(AddVdsCommand.java:315) [engine-bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.internalCanDoAction(CommandBase.java:510) [engine-bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:278) [engine-bll.jar:] > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:336) [engine-bll.jar:] > at org.ovirt.engine.core.bll.Backend.RunAction(Backend.java:294) [engine-bll.jar:] > at sun.reflect.GeneratedMethodAccessor99.invoke(Unknown Source) [:1.7.0_09-icedtea] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_09-icedtea] > at java.lang.reflect.Method.invoke(Method.java:601) [rt.jar:1.7.0_09-icedtea] > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) [jboss-as-ee.jar:7.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:374) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.ovirt.engine.core.utils.ThreadLocalSessionCleanerInterceptor.injectWebContextToThreadLocal(ThreadLocalSessionCleanerInterceptor.java:11) [engine-utils.jar:] > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_09-icedtea] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_09-icedtea] > at java.lang.reflect.Method.invoke(Method.java:601) [rt.jar:1.7.0_09-icedtea] > at org.jboss.as.ee.component.ManagedReferenceLifecycleMethodInterceptorFactory$ManagedReferenceLifecycleMethodInterceptor.processInvocation(ManagedReferenceLifecycleMethodInterceptorFactory.java:123) [jboss-as-ee.jar:7.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:36) [jboss-as-ee.jar:7.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) [jboss-as-ejb3.jar:7.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) [jboss-as-ee.jar:7.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:53) [jboss-as-ejb3.jar:7.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:210) [jboss-as-ejb3.jar:7.1.2.Final-redhat-1] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:362) [jboss-as-ejb3.jar:7.1.2.Final-redhat-1] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:193) [jboss-as-ejb3.jar:7.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [jboss-as-ejb3.jar:7.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:42) [jboss-as-ejb3.jar:7.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) [jboss-as-ejb3.jar:7.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) [jboss-as-ee.jar:7.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) [jboss-as-ee.jar:7.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) [jboss-as-ee.jar:7.1.2.Final-redhat-1] > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:176) [jboss-as-ee.jar:7.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.1.Final-redhat-1] > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) [jboss-as-ee.jar:7.1.2.Final-redhat-1] > at org.ovirt.engine.core.common.interfaces.BackendLocal$$$view9.RunAction(Unknown Source) [engine-common.jar:] > at org.ovirt.engine.ui.frontend.server.gwt.GenericApiGWTServiceImpl.RunAction(GenericApiGWTServiceImpl.java:120) > at sun.reflect.GeneratedMethodAccessor98.invoke(Unknown Source) [:1.7.0_09-icedtea] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_09-icedtea] > at java.lang.reflect.Method.invoke(Method.java:601) [rt.jar:1.7.0_09-icedtea] > at com.google.gwt.rpc.server.RPC.invokeAndStreamResponse(RPC.java:196) > at com.google.gwt.rpc.server.RpcServlet.processCall(RpcServlet.java:161) > at com.google.gwt.rpc.server.RpcServlet.processPost(RpcServlet.java:222) > at com.google.gwt.user.server.rpc.AbstractRemoteServiceServlet.doPost(AbstractRemoteServiceServlet.java:62) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:754) [jboss-servlet-api_3.0_spec.jar:1.0.1.Final-redhat-1] > at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) [jboss-servlet-api_3.0_spec.jar:1.0.1.Final-redhat-1] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:329) > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) > at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:275) > at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:161) > at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:489) > at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:153) > at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:155) > at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) > at org.jboss.web.rewrite.RewriteValve.invoke(RewriteValve.java:466) > at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) > at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:368) > at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:505) > at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:452) > at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:931) > at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_09-icedtea] > >2012-11-08 08:47:42,892 WARN [org.ovirt.engine.core.bll.AddVdsCommand] (ajp-/127.0.0.1:8702-9) [26e64d6d] CanDoAction of action AddVds failed. Reasons:VDS_CANNOT_CONNECT_TO_SERVER,VAR__ACTION__ADD,VAR__TYPE__HOST >2012-11-08 08:49:31,713 INFO [org.ovirt.engine.core.bll.RemoveVmTemplateCommand] (ajp-/127.0.0.1:8702-7) [17d1ac51] Lock Acquired to object EngineLock [exclusiveLocks= key: 54a99fbb-bb42-4f9f-acfd-d495dae03e3a value: TEMPLATE >, sharedLocks= ] >2012-11-08 08:49:31,714 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-7) [17d1ac51] START, IsValidVDSCommand( storagePoolId = f4c98a0b-0578-485f-90f4-743df3d05d7e, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 72e83229 >2012-11-08 08:49:31,715 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-7) [17d1ac51] FINISH, IsValidVDSCommand, return: true, log id: 72e83229 >2012-11-08 08:49:31,751 INFO [org.ovirt.engine.core.bll.RemoveVmTemplateCommand] (ajp-/127.0.0.1:8702-7) [17d1ac51] Running command: RemoveVmTemplateCommand internal: false. Entities affected : ID: 54a99fbb-bb42-4f9f-acfd-d495dae03e3a Type: VmTemplate >2012-11-08 08:49:31,758 INFO [org.ovirt.engine.core.bll.RemoveVmTemplateCommand] (ajp-/127.0.0.1:8702-7) [17d1ac51] Lock freed to object EngineLock [exclusiveLocks= key: 54a99fbb-bb42-4f9f-acfd-d495dae03e3a value: TEMPLATE >, sharedLocks= ] >2012-11-08 08:49:31,758 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.RemoveVMVDSCommand] (ajp-/127.0.0.1:8702-7) [17d1ac51] START, RemoveVMVDSCommand( storagePoolId = f4c98a0b-0578-485f-90f4-743df3d05d7e, ignoreFailoverLimit = false, compatabilityVersion = null, storageDomainId = 00000000-0000-0000-0000-000000000000, vmGuid = 54a99fbb-bb42-4f9f-acfd-d495dae03e3a), log id: 2bafb866 >2012-11-08 08:49:31,797 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.RemoveVMVDSCommand] (ajp-/127.0.0.1:8702-7) [17d1ac51] FINISH, RemoveVMVDSCommand, log id: 2bafb866 >2012-11-08 08:49:31,802 INFO [org.ovirt.engine.core.bll.RemoveAllVmTemplateImageTemplatesCommand] (ajp-/127.0.0.1:8702-7) [17d1ac51] Running command: RemoveAllVmTemplateImageTemplatesCommand internal: true. Entities affected : ID: 54a99fbb-bb42-4f9f-acfd-d495dae03e3a Type: VmTemplate >2012-11-08 08:49:31,817 INFO [org.ovirt.engine.core.bll.RemoveTemplateSnapshotCommand] (ajp-/127.0.0.1:8702-7) [6f1c2123] Running command: RemoveTemplateSnapshotCommand internal: true. Entities affected : ID: 00068b9a-c8e3-4e01-b2c4-963931bfae65 Type: Storage >2012-11-08 08:49:31,819 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] (ajp-/127.0.0.1:8702-7) [6f1c2123] START, DeleteImageGroupVDSCommand( storagePoolId = f4c98a0b-0578-485f-90f4-743df3d05d7e, ignoreFailoverLimit = false, compatabilityVersion = 3.1, storageDomainId = 00068b9a-c8e3-4e01-b2c4-963931bfae65, imageGroupId = e95ccbb4-5049-43fd-b367-8658058f36ee, postZeros = false, forceDelete = false), log id: 303207b1 >2012-11-08 08:49:32,056 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] (ajp-/127.0.0.1:8702-7) [6f1c2123] FINISH, DeleteImageGroupVDSCommand, log id: 303207b1 >2012-11-08 08:49:32,060 INFO [org.ovirt.engine.core.bll.EntityAsyncTask] (ajp-/127.0.0.1:8702-7) [6f1c2123] EntityAsyncTask::Adding EntityMultiAsyncTasks object for entity 54a99fbb-bb42-4f9f-acfd-d495dae03e3a >2012-11-08 08:49:32,060 INFO [org.ovirt.engine.core.bll.EntityMultiAsyncTasks] (ajp-/127.0.0.1:8702-7) [6f1c2123] EntityMultiAsyncTasks::AttachTask: Attaching task d9662824-9133-4a47-8929-84b54ba82304 to entity 54a99fbb-bb42-4f9f-acfd-d495dae03e3a. >2012-11-08 08:49:32,061 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskUtils] (ajp-/127.0.0.1:8702-7) [6f1c2123] Adding task d9662824-9133-4a47-8929-84b54ba82304 to DataBase >2012-11-08 08:49:32,067 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (ajp-/127.0.0.1:8702-7) [6f1c2123] Adding task d9662824-9133-4a47-8929-84b54ba82304 (Parent Command RemoveVmTemplate, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters), polling hasn't started yet.. >2012-11-08 08:49:32,084 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (ajp-/127.0.0.1:8702-7) [6f1c2123] BaseAsyncTask::StartPollingTask: Starting to poll task d9662824-9133-4a47-8929-84b54ba82304. >2012-11-08 08:49:32,085 INFO [org.ovirt.engine.core.bll.EntityMultiAsyncTasks] (ajp-/127.0.0.1:8702-7) [6f1c2123] EntityMultiAsyncTasks::StartPollingTask: Current Action Type for entity 54a99fbb-bb42-4f9f-acfd-d495dae03e3a is RemoveVmTemplate (determined by task d9662824-9133-4a47-8929-84b54ba82304) >2012-11-08 08:49:38,974 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-39) Polling and updating Async Tasks: 1 tasks, 1 tasks to poll now >2012-11-08 08:49:38,995 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (QuartzScheduler_Worker-39) SPMAsyncTask::PollTask: Polling task d9662824-9133-4a47-8929-84b54ba82304 (Parent Command RemoveVmTemplate, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status finished, result 'success'. >2012-11-08 08:49:38,998 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (QuartzScheduler_Worker-39) BaseAsyncTask::OnTaskEndSuccess: Task d9662824-9133-4a47-8929-84b54ba82304 (Parent Command RemoveVmTemplate, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended successfully. >2012-11-08 08:49:38,999 INFO [org.ovirt.engine.core.bll.EntityAsyncTask] (QuartzScheduler_Worker-39) EntityAsyncTask::EndActionIfNecessary: All tasks of entity 54a99fbb-bb42-4f9f-acfd-d495dae03e3a has ended -> executing EndAction >2012-11-08 08:49:38,999 INFO [org.ovirt.engine.core.bll.EntityAsyncTask] (QuartzScheduler_Worker-39) EntityAsyncTask::EndAction: Ending action for 1 tasks (entity ID: 54a99fbb-bb42-4f9f-acfd-d495dae03e3a): calling EndAction for action type RemoveVmTemplate. >2012-11-08 08:49:38,999 INFO [org.ovirt.engine.core.bll.EntityAsyncTask] (pool-4-thread-10) EntityAsyncTask::EndCommandAction [within thread]context: Attempting to EndAction RemoveVmTemplate >2012-11-08 08:49:39,007 INFO [org.ovirt.engine.core.bll.RemoveVmTemplateCommand] (pool-4-thread-10) [17d1ac51] Ending command successfully: org.ovirt.engine.core.bll.RemoveVmTemplateCommand >2012-11-08 08:49:39,028 INFO [org.ovirt.engine.core.bll.EntityAsyncTask] (pool-4-thread-10) EntityAsyncTask::HandleEndActionResult [within thread]: EndAction for action type RemoveVmTemplate completed, handling the result. >2012-11-08 08:49:39,028 INFO [org.ovirt.engine.core.bll.EntityAsyncTask] (pool-4-thread-10) EntityAsyncTask::HandleEndActionResult [within thread]: EndAction for action type RemoveVmTemplate succeeded, clearing tasks. >2012-11-08 08:49:39,030 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (pool-4-thread-10) SPMAsyncTask::ClearAsyncTask: Attempting to clear task d9662824-9133-4a47-8929-84b54ba82304 >2012-11-08 08:49:39,031 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (pool-4-thread-10) START, SPMClearTaskVDSCommand( storagePoolId = f4c98a0b-0578-485f-90f4-743df3d05d7e, ignoreFailoverLimit = false, compatabilityVersion = null, taskId = d9662824-9133-4a47-8929-84b54ba82304), log id: 1941d98 >2012-11-08 08:49:39,033 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (pool-4-thread-10) START, HSMClearTaskVDSCommand(HostName = rhs-arch-srv2.lab.eng.blr.redhat.com, HostId = cf017816-29a2-11e2-949e-52540035ed48, taskId=d9662824-9133-4a47-8929-84b54ba82304), log id: 2ed30add >2012-11-08 08:49:39,073 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (pool-4-thread-10) FINISH, HSMClearTaskVDSCommand, log id: 2ed30add >2012-11-08 08:49:39,074 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (pool-4-thread-10) FINISH, SPMClearTaskVDSCommand, log id: 1941d98 >2012-11-08 08:49:39,076 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (pool-4-thread-10) BaseAsyncTask::RemoveTaskFromDB: Removed task d9662824-9133-4a47-8929-84b54ba82304 from DataBase >2012-11-08 08:49:39,076 INFO [org.ovirt.engine.core.bll.EntityAsyncTask] (pool-4-thread-10) EntityAsyncTask::HandleEndActionResult [within thread]: Removing EntityMultiAsyncTasks object for entity 54a99fbb-bb42-4f9f-acfd-d495dae03e3a >2012-11-08 08:50:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-82) Autorecovering hosts is disabled, skipping >2012-11-08 08:50:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-82) Autorecovering storage domains is disabled, skipping >2012-11-08 08:50:34,048 INFO [org.ovirt.engine.core.bll.SetVmTicketCommand] (ajp-/127.0.0.1:8702-15) [6019e8f8] Running command: SetVmTicketCommand internal: false. Entities affected : ID: 68655de5-7bd9-4bdc-8628-3b2069ec312f Type: VM >2012-11-08 08:50:34,068 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp-/127.0.0.1:8702-15) [6019e8f8] START, SetVmTicketVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=68655de5-7bd9-4bdc-8628-3b2069ec312f, ticket=sOMyRdkPkb4U, validTime=120,m userName=admin@internal, userId=fdfc627c-d875-11e0-90f0-83df133b58cc), log id: 218a7c95 >2012-11-08 08:50:34,100 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp-/127.0.0.1:8702-15) [6019e8f8] FINISH, SetVmTicketVDSCommand, log id: 218a7c95 >2012-11-08 08:50:56,399 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-69) Setting new tasks map. The map contains now 0 tasks >2012-11-08 08:50:56,399 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-69) Cleared all tasks of pool f4c98a0b-0578-485f-90f4-743df3d05d7e. >2012-11-08 08:50:56,402 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-69) Could not find vds that is spm and non-operational. >2012-11-08 08:51:12,748 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-10) [671dff4a] START, IsValidVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 1c059a40 >2012-11-08 08:51:12,749 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-10) [671dff4a] FINISH, IsValidVDSCommand, return: true, log id: 1c059a40 >2012-11-08 08:51:12,761 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp-/127.0.0.1:8702-10) [671dff4a] START, IsVmDuringInitiatingVDSCommand( vmId = 3d766d10-58ab-4306-b5c4-7d109c7b6c46), log id: 7fa09257 >2012-11-08 08:51:12,761 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp-/127.0.0.1:8702-10) [671dff4a] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 7fa09257 >2012-11-08 08:51:12,813 INFO [org.ovirt.engine.core.bll.RunVmOnceCommand] (ajp-/127.0.0.1:8702-10) [671dff4a] Running command: RunVmOnceCommand internal: false. Entities affected : ID: 3d766d10-58ab-4306-b5c4-7d109c7b6c46 Type: VM >2012-11-08 08:51:12,871 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsoPrefixVDSCommand] (ajp-/127.0.0.1:8702-10) [671dff4a] START, IsoPrefixVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 435e864f >2012-11-08 08:51:12,872 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsoPrefixVDSCommand] (ajp-/127.0.0.1:8702-10) [671dff4a] FINISH, IsoPrefixVDSCommand, return: /rhev/data-center/3a17dcac-a6e3-4563-a4b0-f27d89b68dd6/7a268ac7-857a-4c53-966d-247deb509161/images/11111111-1111-1111-1111-111111111111, log id: 435e864f >2012-11-08 08:51:12,887 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-10) [671dff4a] START, CreateVmVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=3d766d10-58ab-4306-b5c4-7d109c7b6c46, vm=org.ovirt.engine.core.common.businessentities.VM@1d060f3d), log id: 72c3cf16 >2012-11-08 08:51:12,900 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8702-10) [671dff4a] START, CreateVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=3d766d10-58ab-4306-b5c4-7d109c7b6c46, vm=org.ovirt.engine.core.common.businessentities.VM@1d060f3d), log id: c445c41 >2012-11-08 08:51:12,953 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8702-10) [671dff4a] org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand spiceSslCipherSuite=DEFAULT,memSize=512,kvmEnable=true,smp=1,emulatedMachine=rhel6.3.0,vmType=kvm,keyboardLayout=en-us,nice=0,display=qxl,smpCoresPerSocket=1,spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard,timeOffset=-43200,transparentHugePages=true,vmId=3d766d10-58ab-4306-b5c4-7d109c7b6c46,devices=[Ljava.util.Map;@4f2e19d9,acpiEnable=true,vmName=b5,cpuType=Nehalem,custom={} >2012-11-08 08:51:12,953 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8702-10) [671dff4a] FINISH, CreateVDSCommand, log id: c445c41 >2012-11-08 08:51:12,953 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-10) [671dff4a] IncreasePendingVms::CreateVmIncreasing vds rhs-gp-srv11.lab.eng.blr.redhat.com pending vcpu count, now 1. Vm: b5 >2012-11-08 08:51:12,955 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-10) [671dff4a] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 72c3cf16 >2012-11-08 08:51:13,458 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (QuartzScheduler_Worker-23) START, FullListVdsCommand( HostId = 96739be4-2990-11e2-8577-52540035ed48, vds=null, vmIds=[3d766d10-58ab-4306-b5c4-7d109c7b6c46]), log id: 7ba5e765 >2012-11-08 08:51:13,478 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (QuartzScheduler_Worker-23) FINISH, FullListVdsCommand, return: [Lorg.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcStruct;@48fad74e, log id: 7ba5e765 >2012-11-08 08:51:13,481 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-23) Recieved a qxl Device without an address when processing VM 3d766d10-58ab-4306-b5c4-7d109c7b6c46 devices, skipping device: {specParams={vram=65536}, device=qxl, type=video, deviceId=383a3ece-79f1-4834-bec1-3f7e4031cfa0} >2012-11-08 08:51:13,481 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-23) Recieved a bridge Device without an address when processing VM 3d766d10-58ab-4306-b5c4-7d109c7b6c46 devices, skipping device: {bootOrder=2, nicModel=pv, specParams={}, device=bridge, macAddr=00:1a:4a:46:22:63, type=interface, deviceId=fae63f76-21f7-49d3-86d8-c7edb3a942d8, filter=vdsm-no-mac-spoofing, network=rhevm} >2012-11-08 08:51:13,481 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-23) Recieved a memballoon Device without an address when processing VM 3d766d10-58ab-4306-b5c4-7d109c7b6c46 devices, skipping device: {specParams={model=virtio}, device=memballoon, type=balloon, deviceId=9397b270-956e-4951-83b0-ac16735ae445} >2012-11-08 08:51:13,481 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-23) Recieved a cdrom Device without an address when processing VM 3d766d10-58ab-4306-b5c4-7d109c7b6c46 devices, skipping device: {shared=false, iface=ide, index=2, specParams={path=}, device=cdrom, path=, type=disk, readonly=true, deviceId=43ac7ecc-5ac0-4506-a8df-bf7e6bd497aa} >2012-11-08 08:51:13,481 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-23) Recieved a disk Device without an address when processing VM 3d766d10-58ab-4306-b5c4-7d109c7b6c46 devices, skipping device: {shared=false, index=0, volumeID=c5444624-5b51-4a11-a597-bb21e7285b85, propagateErrors=off, format=raw, type=disk, truesize=5368709120, reqsize=0, iface=virtio, bootOrder=1, volumeChain=[Ljava.lang.Object;@2b1a0bff, imageID=f6d26f54-5fe4-4c74-8660-ae059dafd9a6, domainID=a02e96be-cf4c-4d0f-8368-fd9d4aa659bf, specParams={}, optional=false, path=/rhev/data-center/3a17dcac-a6e3-4563-a4b0-f27d89b68dd6/a02e96be-cf4c-4d0f-8368-fd9d4aa659bf/images/f6d26f54-5fe4-4c74-8660-ae059dafd9a6/c5444624-5b51-4a11-a597-bb21e7285b85, device=disk, poolID=3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, readonly=false, deviceId=f6d26f54-5fe4-4c74-8660-ae059dafd9a6, apparentsize=5368709120} >2012-11-08 08:51:15,601 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (QuartzScheduler_Worker-21) START, DestroyVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=3d766d10-58ab-4306-b5c4-7d109c7b6c46, force=false, secondsToWait=0, gracefully=false), log id: 6a6ce537 >2012-11-08 08:51:15,662 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (QuartzScheduler_Worker-21) FINISH, DestroyVDSCommand, log id: 6a6ce537 >2012-11-08 08:51:15,681 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-21) Running on vds during rerun failed vm: null >2012-11-08 08:51:15,682 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-21) vm b5 running in db and not running in vds - add to rerun treatment. vds rhs-gp-srv11.lab.eng.blr.redhat.com >2012-11-08 08:51:15,686 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-21) Rerun vm 3d766d10-58ab-4306-b5c4-7d109c7b6c46. Called from vds rhs-gp-srv11.lab.eng.blr.redhat.com >2012-11-08 08:51:15,714 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsDynamicDataVDSCommand] (pool-4-thread-45) START, UpdateVdsDynamicDataVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vdsDynamic=org.ovirt.engine.core.common.businessentities.VdsDynamic@c73b9a9), log id: 7131264 >2012-11-08 08:51:15,715 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsDynamicDataVDSCommand] (pool-4-thread-45) FINISH, UpdateVdsDynamicDataVDSCommand, log id: 7131264 >2012-11-08 08:51:15,730 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (pool-4-thread-45) START, IsValidVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 366e8306 >2012-11-08 08:51:15,731 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (pool-4-thread-45) FINISH, IsValidVDSCommand, return: true, log id: 366e8306 >2012-11-08 08:51:15,764 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (pool-4-thread-45) START, IsVmDuringInitiatingVDSCommand( vmId = 3d766d10-58ab-4306-b5c4-7d109c7b6c46), log id: 4c0f2bf7 >2012-11-08 08:51:15,764 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (pool-4-thread-45) FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 4c0f2bf7 >2012-11-08 08:51:15,798 INFO [org.ovirt.engine.core.bll.VdsSelector] (pool-4-thread-45) VDS rhs-gp-srv11.lab.eng.blr.redhat.com 96739be4-2990-11e2-8577-52540035ed48 is the same host the VM is currently running on VDS rhs-gp-srv4.lab.eng.blr.redhat.com d2bd32d4-2992-11e2-ab59-52540035ed48 is not in up status or belongs to the VM's cluster VDS rhs-arch-srv2.lab.eng.blr.redhat.com cf017816-29a2-11e2-949e-52540035ed48 is not in up status or belongs to the VM's cluster >2012-11-08 08:51:15,798 WARN [org.ovirt.engine.core.bll.RunVmOnceCommand] (pool-4-thread-45) CanDoAction of action RunVmOnce failed. Reasons:VAR__ACTION__RUN,VAR__TYPE__VM,$VmName b5,VAR__ACTION__RUN,VAR__TYPE__VM,ACTION_TYPE_FAILED_VDS_VM_CLUSTER >2012-11-08 08:51:24,081 INFO [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (ajp-/127.0.0.1:8702-15) SSH execute 10.70.36.16 'IDFILE=/etc/vdsm/vdsm.id; if [ -r "${IDFILE}" ]; then cat "${IDFILE}"; else UUID="$(dmidecode -s system-uuid 2> /dev/null | sed -e 's/.*Not.*//' )"; if [ -z "${UUID}" ]; then UUID="$(uuidgen 2> /dev/null)" && mkdir -p "$(dirname "${IDFILE}")" && echo "${UUID}" > "${IDFILE}" && chmod 0644 "${IDFILE}"; fi; [ -n "${UUID}" ] && echo "${UUID}"; fi' >2012-11-08 08:51:24,218 INFO [org.ovirt.engine.core.bll.AddVdsCommand] (ajp-/127.0.0.1:8702-15) [5d6d9a09] Running command: AddVdsCommand internal: false. Entities affected : ID: c04832b4-2976-11e2-b6fb-52540035ed48 Type: VdsGroups >2012-11-08 08:51:24,226 INFO [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp-/127.0.0.1:8702-15) [8e47481] Running command: AddVdsSpmIdCommand internal: true. Entities affected : ID: 62704566-29ab-11e2-82bf-52540035ed48 Type: VDS >2012-11-08 08:51:24,231 INFO [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp-/127.0.0.1:8702-15) [8e47481] Lock freed to object EngineLock [exclusiveLocks= key: 1eaedbbb-1c0e-4028-9b0e-adac7e07d25f value: REGISTER_VDS >, sharedLocks= ] >2012-11-08 08:51:24,231 ERROR [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp-/127.0.0.1:8702-15) [8e47481] Cannot get vdsManager for vdsid=62704566-29ab-11e2-82bf-52540035ed48 >2012-11-08 08:51:24,231 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (ajp-/127.0.0.1:8702-15) [8e47481] START, RemoveVdsVDSCommand(HostName = rhs-gp-srv12.lab.eng.blr.redhat.com, HostId = 62704566-29ab-11e2-82bf-52540035ed48), log id: 265f43b1 >2012-11-08 08:51:24,231 ERROR [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp-/127.0.0.1:8702-15) [8e47481] Cannot get vdsManager for vdsid=62704566-29ab-11e2-82bf-52540035ed48 >2012-11-08 08:51:24,232 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (ajp-/127.0.0.1:8702-15) [8e47481] FINISH, RemoveVdsVDSCommand, log id: 265f43b1 >2012-11-08 08:51:24,232 ERROR [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp-/127.0.0.1:8702-15) [8e47481] Cannot get vdsManager for vdsid=62704566-29ab-11e2-82bf-52540035ed48 >2012-11-08 08:51:24,232 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-15) [8e47481] START, AddVdsVDSCommand(HostName = rhs-gp-srv12.lab.eng.blr.redhat.com, HostId = 62704566-29ab-11e2-82bf-52540035ed48), log id: 4a8bd811 >2012-11-08 08:51:24,232 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-15) [8e47481] AddVds - entered , starting logic to add VDS 62704566-29ab-11e2-82bf-52540035ed48 >2012-11-08 08:51:24,234 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-15) [8e47481] AddVds - VDS 62704566-29ab-11e2-82bf-52540035ed48 was added, will try to add it to the resource manager >2012-11-08 08:51:24,235 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ajp-/127.0.0.1:8702-15) [8e47481] Eneterd VdsManager:constructor >2012-11-08 08:51:24,235 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ajp-/127.0.0.1:8702-15) [8e47481] vdsBroker(10.70.36.16,54,321) >2012-11-08 08:51:24,235 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp-/127.0.0.1:8702-15) [8e47481] ResourceManager::AddVds - VDS 62704566-29ab-11e2-82bf-52540035ed48 was added to the Resource Manager >2012-11-08 08:51:24,235 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-15) [8e47481] FINISH, AddVdsVDSCommand, log id: 4a8bd811 >2012-11-08 08:51:24,250 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-4-thread-45) [46a65120] Running command: InstallVdsCommand internal: true. Entities affected : ID: 62704566-29ab-11e2-82bf-52540035ed48 Type: VDS >2012-11-08 08:51:24,251 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-4-thread-45) [46a65120] Before Installation pool-4-thread-45 >2012-11-08 08:51:24,251 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-45) [46a65120] Installation of 10.70.36.16. Executing installation stage. (Stage: Starting Host installation) >2012-11-08 08:51:24,251 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-45) [46a65120] Installation of 10.70.36.16. Executing installation stage. (Stage: Connecting to Host) >2012-11-08 08:51:24,314 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-45) [46a65120] Installation of 10.70.36.16. Received message: <BSTRAP component='RHEV_INSTALL' status='OK' message='Connected to Host 10.70.36.16 with SSH key fingerprint: d8:e6:4c:57:0c:54:e1:5a:31:d9:fa:80:44:2a:ac:7f'/>. FYI. (Stage: Connecting to Host) >2012-11-08 08:51:24,343 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-45) [46a65120] Installation of 10.70.36.16. Successfully connected to server ssh. (Stage: Connecting to Host) >2012-11-08 08:51:24,343 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-45) [46a65120] Installation of 10.70.36.16. Executing installation stage. (Stage: Get the unique vds id) >2012-11-08 08:51:24,343 INFO [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-4-thread-45) SSH execute 10.70.36.16 'IDFILE=/etc/vdsm/vdsm.id; if [ -r "${IDFILE}" ]; then cat "${IDFILE}"; else UUID="$(dmidecode -s system-uuid 2> /dev/null | sed -e 's/.*Not.*//' )"; if [ -z "${UUID}" ]; then UUID="$(uuidgen 2> /dev/null)" && mkdir -p "$(dirname "${IDFILE}")" && echo "${UUID}" > "${IDFILE}" && chmod 0644 "${IDFILE}"; fi; [ -n "${UUID}" ] && echo "${UUID}"; fi' >2012-11-08 08:51:24,453 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-79) Installation of 10.70.36.16. Received message: 00000000-0000-0000-0000-0025902F723C. FYI. (Stage: Get the unique vds id) >2012-11-08 08:51:24,455 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-79) Installation of 10.70.36.16. Assigning unique id 00000000-0000-0000-0000-0025902F723C to Host. (Stage: Get the unique vds id) >2012-11-08 08:51:24,458 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-45) [46a65120] Installation of 10.70.36.16. Executing installation stage. (Stage: Upload Installation script to Host) >2012-11-08 08:51:24,458 INFO [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-4-thread-45) SSH execute 10.70.36.16 'cat > '/tmp/ovirt-id_rsa_0cb47953-de57-4054-b6f9-e09a65dd4fb9'' >2012-11-08 08:51:24,541 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-45) [46a65120] Installation of 10.70.36.16. Executing installation stage. (Stage: Running first installation script on Host) >2012-11-08 08:51:24,541 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-45) [46a65120] Installation of 10.70.36.16. Sending SSH Command umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar -C "${MYTMP}" --no-same-permissions -o -x && "${MYTMP}"/setup -c 'ssl=true;management_port=54321' -O 'lab.eng.blr.redhat.com' -t 2012-11-08T13:51:24 -S /tmp/ovirt-id_rsa_0cb47953-de57-4054-b6f9-e09a65dd4fb9 -p 80 -b -B rhevm http://guido.lab.eng.blr.redhat.com:80/Components/vds/ http://guido.lab.eng.blr.redhat.com:80/Components/vds/ 10.70.36.16 0cb47953-de57-4054-b6f9-e09a65dd4fb9 False < /var/cache/ovirt-engine/vdsm-bootstrap-2.tar. (Stage: Running first installation script on Host) >2012-11-08 08:51:24,542 INFO [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-4-thread-45) SSH execute 10.70.36.16 'umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar -C "${MYTMP}" --no-same-permissions -o -x && "${MYTMP}"/setup -c 'ssl=true;management_port=54321' -O 'lab.eng.blr.redhat.com' -t 2012-11-08T13:51:24 -S /tmp/ovirt-id_rsa_0cb47953-de57-4054-b6f9-e09a65dd4fb9 -p 80 -b -B rhevm http://guido.lab.eng.blr.redhat.com:80/Components/vds/ http://guido.lab.eng.blr.redhat.com:80/Components/vds/ 10.70.36.16 0cb47953-de57-4054-b6f9-e09a65dd4fb9 False' >2012-11-08 08:51:24,648 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-79) Installation of 10.70.36.16. Received message: <BSTRAP component='INSTALLER' status='OK' message='Test platform succeeded'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:51:25,789 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-79) Installation of 10.70.36.16. Received message: <BSTRAP component='SET_SYSTEM_TIME' status='OK' message='setSystemTime ended successfully'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:51:26,236 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-49) Initializing Host: rhs-gp-srv12.lab.eng.blr.redhat.com >2012-11-08 08:51:28,687 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-79) Installation of 10.70.36.16. Received message: <BSTRAP component='RHN_REGISTRATION' status='OK' message='Host properly registered with RHN/Satellite.'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:51:28,858 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-79) Installation of 10.70.36.16. Received message: <BSTRAP component='VDSM_MAJOR_VER' status='OK' message='Available VDSM matches requirements'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:51:28,862 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-79) Installation of 10.70.36.16. Received message: <BSTRAP component='VT_SVM' status='OK' processor='Intel' message='Server supports virtualization'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:51:28,867 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-79) Installation of 10.70.36.16. Received message: <BSTRAP component='OS' status='OK' type='RHEL6' message='Supported platform version'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:51:28,872 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-79) Installation of 10.70.36.16. Received message: <BSTRAP component='KERNEL' status='OK' version='279' message='Supported kernel version: 279'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:51:28,960 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-79) Installation of 10.70.36.16. Received message: <BSTRAP component='REQ PACKAGES' status='OK' result='SDL.x86_64' message='SDL-1.2.14-3.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:51:28,975 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-79) Installation of 10.70.36.16. Received message: <BSTRAP component='REQ PACKAGES' status='OK' result='bridge-utils.x86_64' message='bridge-utils-1.2-9.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:51:28,989 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-79) Installation of 10.70.36.16. Received message: <BSTRAP component='REQ PACKAGES' status='OK' result='mesa-libGLU.x86_64' message='mesa-libGLU-7.11-5.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:51:29,004 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-79) Installation of 10.70.36.16. Received message: <BSTRAP component='REQ PACKAGES' status='OK' result='openssl.x86_64' message='openssl-1.0.0-25.el6_3.1.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:51:29,019 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-79) Installation of 10.70.36.16. Received message: <BSTRAP component='REQ PACKAGES' status='OK' result='m2crypto.x86_64' message='m2crypto-0.20.2-9.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:51:29,033 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-79) Installation of 10.70.36.16. Received message: <BSTRAP component='REQ PACKAGES' status='OK' result='rsync.x86_64' message='rsync-3.0.6-9.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:51:29,046 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-79) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='WARN' result='qemu-kvm' message='package qemu-kvm is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:51:29,059 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-79) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='WARN' result='qemu-kvm-tools' message='package qemu-kvm-tools is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:51:29,073 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-79) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='(~vdsm~, ~vdsm-cli~)' message='vdsm-4.9.6-41.0.el6_3.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:51:29,087 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-79) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='libjpeg' message='libjpeg-6b-46.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:51:29,101 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-79) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='spice-server' message='spice-server-0.10.1-10.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:51:29,115 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-79) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='pixman' message='pixman-0.18.4-1.el6_0.1.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:51:29,129 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-79) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='seabios' message='seabios-0.6.1.2-19.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:51:29,142 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-79) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='WARN' result='qemu-img' message='package qemu-img is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:51:29,156 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-79) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='fence-agents' message='fence-agents-3.1.5-17.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:51:29,170 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-79) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='libselinux-python' message='libselinux-python-2.0.94-5.3.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:51:29,184 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-79) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='sanlock' message='sanlock-2.3-4.el6_3.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:51:29,197 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-79) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='sanlock-python' message='sanlock-python-2.3-4.el6_3.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:51:46,288 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-11) [50ddbcbe] START, IsValidVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 2f5715f6 >2012-11-08 08:51:46,289 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-11) [50ddbcbe] FINISH, IsValidVDSCommand, return: true, log id: 2f5715f6 >2012-11-08 08:51:46,300 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp-/127.0.0.1:8702-11) [50ddbcbe] START, IsVmDuringInitiatingVDSCommand( vmId = 3d766d10-58ab-4306-b5c4-7d109c7b6c46), log id: dfd8da >2012-11-08 08:51:46,300 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp-/127.0.0.1:8702-11) [50ddbcbe] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: dfd8da >2012-11-08 08:51:46,338 INFO [org.ovirt.engine.core.bll.RunVmOnceCommand] (ajp-/127.0.0.1:8702-11) [50ddbcbe] Running command: RunVmOnceCommand internal: false. Entities affected : ID: 3d766d10-58ab-4306-b5c4-7d109c7b6c46 Type: VM >2012-11-08 08:51:46,384 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsoPrefixVDSCommand] (ajp-/127.0.0.1:8702-11) [50ddbcbe] START, IsoPrefixVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 52e6961f >2012-11-08 08:51:46,384 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsoPrefixVDSCommand] (ajp-/127.0.0.1:8702-11) [50ddbcbe] FINISH, IsoPrefixVDSCommand, return: /rhev/data-center/3a17dcac-a6e3-4563-a4b0-f27d89b68dd6/7a268ac7-857a-4c53-966d-247deb509161/images/11111111-1111-1111-1111-111111111111, log id: 52e6961f >2012-11-08 08:51:46,402 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-11) [50ddbcbe] START, CreateVmVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=3d766d10-58ab-4306-b5c4-7d109c7b6c46, vm=org.ovirt.engine.core.common.businessentities.VM@1282179e), log id: 2089f7be >2012-11-08 08:51:46,413 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8702-11) [50ddbcbe] START, CreateVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=3d766d10-58ab-4306-b5c4-7d109c7b6c46, vm=org.ovirt.engine.core.common.businessentities.VM@1282179e), log id: 7f0e747 >2012-11-08 08:51:46,464 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8702-11) [50ddbcbe] org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand spiceSslCipherSuite=DEFAULT,memSize=512,kvmEnable=true,smp=1,emulatedMachine=rhel6.3.0,vmType=kvm,keyboardLayout=en-us,nice=0,display=qxl,smpCoresPerSocket=1,spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard,timeOffset=-43200,transparentHugePages=true,vmId=3d766d10-58ab-4306-b5c4-7d109c7b6c46,devices=[Ljava.util.Map;@54438e77,acpiEnable=true,vmName=b5,cpuType=Nehalem,custom={} >2012-11-08 08:51:46,464 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8702-11) [50ddbcbe] FINISH, CreateVDSCommand, log id: 7f0e747 >2012-11-08 08:51:46,465 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-11) [50ddbcbe] IncreasePendingVms::CreateVmIncreasing vds rhs-gp-srv11.lab.eng.blr.redhat.com pending vcpu count, now 1. Vm: b5 >2012-11-08 08:51:46,467 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-11) [50ddbcbe] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 2089f7be >2012-11-08 08:51:48,687 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (QuartzScheduler_Worker-1) START, DestroyVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=3d766d10-58ab-4306-b5c4-7d109c7b6c46, force=false, secondsToWait=0, gracefully=false), log id: 5bd13019 >2012-11-08 08:51:48,746 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (QuartzScheduler_Worker-1) FINISH, DestroyVDSCommand, log id: 5bd13019 >2012-11-08 08:51:48,766 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-1) Running on vds during rerun failed vm: null >2012-11-08 08:51:48,767 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-1) vm b5 running in db and not running in vds - add to rerun treatment. vds rhs-gp-srv11.lab.eng.blr.redhat.com >2012-11-08 08:51:48,771 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-1) Rerun vm 3d766d10-58ab-4306-b5c4-7d109c7b6c46. Called from vds rhs-gp-srv11.lab.eng.blr.redhat.com >2012-11-08 08:51:48,795 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsDynamicDataVDSCommand] (pool-4-thread-10) START, UpdateVdsDynamicDataVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vdsDynamic=org.ovirt.engine.core.common.businessentities.VdsDynamic@4bf11779), log id: 788ee7e4 >2012-11-08 08:51:48,797 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsDynamicDataVDSCommand] (pool-4-thread-10) FINISH, UpdateVdsDynamicDataVDSCommand, log id: 788ee7e4 >2012-11-08 08:51:48,810 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (pool-4-thread-10) START, IsValidVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 4a9140ad >2012-11-08 08:51:48,811 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (pool-4-thread-10) FINISH, IsValidVDSCommand, return: true, log id: 4a9140ad >2012-11-08 08:51:48,823 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (pool-4-thread-10) START, IsVmDuringInitiatingVDSCommand( vmId = 3d766d10-58ab-4306-b5c4-7d109c7b6c46), log id: 5ccb0a82 >2012-11-08 08:51:48,823 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (pool-4-thread-10) FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 5ccb0a82 >2012-11-08 08:51:48,845 INFO [org.ovirt.engine.core.bll.VdsSelector] (pool-4-thread-10) VDS rhs-gp-srv12.lab.eng.blr.redhat.com 62704566-29ab-11e2-82bf-52540035ed48 is not in up status or belongs to the VM's cluster VDS rhs-gp-srv11.lab.eng.blr.redhat.com 96739be4-2990-11e2-8577-52540035ed48 is the same host the VM is currently running on VDS rhs-gp-srv4.lab.eng.blr.redhat.com d2bd32d4-2992-11e2-ab59-52540035ed48 is not in up status or belongs to the VM's cluster VDS rhs-arch-srv2.lab.eng.blr.redhat.com cf017816-29a2-11e2-949e-52540035ed48 is not in up status or belongs to the VM's cluster >2012-11-08 08:51:48,845 WARN [org.ovirt.engine.core.bll.RunVmOnceCommand] (pool-4-thread-10) CanDoAction of action RunVmOnce failed. Reasons:VAR__ACTION__RUN,VAR__TYPE__VM,$VmName b5,VAR__ACTION__RUN,VAR__TYPE__VM,ACTION_TYPE_FAILED_VDS_VM_CLUSTER >2012-11-08 08:51:59,802 WARN [org.ovirt.engine.core.bll.AddVmTemplateCommand] (ajp-/127.0.0.1:8702-11) [2ed81eaf] CanDoAction of action AddVmTemplate failed. Reasons:VAR__ACTION__ADD,VAR__TYPE__VM_TEMPLATE,VMT_CANNOT_CREATE_TEMPLATE_FROM_DOWN_VM >2012-11-08 08:51:59,843 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (ajp-/127.0.0.1:8702-11) Operation Failed: [Cannot add Template. VM still running, stop the VM and try again.] >2012-11-08 08:52:27,105 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-15) [342683bd] START, IsValidVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 21208ab0 >2012-11-08 08:52:27,106 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-15) [342683bd] FINISH, IsValidVDSCommand, return: true, log id: 21208ab0 >2012-11-08 08:52:27,121 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp-/127.0.0.1:8702-15) [342683bd] START, IsVmDuringInitiatingVDSCommand( vmId = 3d766d10-58ab-4306-b5c4-7d109c7b6c46), log id: 4503418b >2012-11-08 08:52:27,121 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp-/127.0.0.1:8702-15) [342683bd] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 4503418b >2012-11-08 08:52:27,167 INFO [org.ovirt.engine.core.bll.RunVmOnceCommand] (ajp-/127.0.0.1:8702-15) [342683bd] Running command: RunVmOnceCommand internal: false. Entities affected : ID: 3d766d10-58ab-4306-b5c4-7d109c7b6c46 Type: VM >2012-11-08 08:52:27,216 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsoPrefixVDSCommand] (ajp-/127.0.0.1:8702-15) [342683bd] START, IsoPrefixVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 1805e4cd >2012-11-08 08:52:27,216 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsoPrefixVDSCommand] (ajp-/127.0.0.1:8702-15) [342683bd] FINISH, IsoPrefixVDSCommand, return: /rhev/data-center/3a17dcac-a6e3-4563-a4b0-f27d89b68dd6/7a268ac7-857a-4c53-966d-247deb509161/images/11111111-1111-1111-1111-111111111111, log id: 1805e4cd >2012-11-08 08:52:27,231 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-15) [342683bd] START, CreateVmVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=3d766d10-58ab-4306-b5c4-7d109c7b6c46, vm=org.ovirt.engine.core.common.businessentities.VM@35963159), log id: 2af05b0d >2012-11-08 08:52:27,246 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8702-15) [342683bd] START, CreateVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=3d766d10-58ab-4306-b5c4-7d109c7b6c46, vm=org.ovirt.engine.core.common.businessentities.VM@35963159), log id: 250c87e2 >2012-11-08 08:52:27,300 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8702-15) [342683bd] org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand spiceSslCipherSuite=DEFAULT,memSize=512,kvmEnable=true,smp=1,emulatedMachine=rhel6.3.0,vmType=kvm,keyboardLayout=en-us,nice=0,display=qxl,smpCoresPerSocket=1,spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard,timeOffset=-43200,transparentHugePages=true,vmId=3d766d10-58ab-4306-b5c4-7d109c7b6c46,devices=[Ljava.util.Map;@54cb79fa,acpiEnable=true,vmName=b5,cpuType=Nehalem,custom={} >2012-11-08 08:52:27,300 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8702-15) [342683bd] FINISH, CreateVDSCommand, log id: 250c87e2 >2012-11-08 08:52:27,300 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-15) [342683bd] IncreasePendingVms::CreateVmIncreasing vds rhs-gp-srv11.lab.eng.blr.redhat.com pending vcpu count, now 1. Vm: b5 >2012-11-08 08:52:27,302 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-15) [342683bd] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 2af05b0d >2012-11-08 08:52:30,051 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (QuartzScheduler_Worker-3) START, DestroyVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=3d766d10-58ab-4306-b5c4-7d109c7b6c46, force=false, secondsToWait=0, gracefully=false), log id: 89a55ac >2012-11-08 08:52:30,112 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (QuartzScheduler_Worker-3) FINISH, DestroyVDSCommand, log id: 89a55ac >2012-11-08 08:52:30,129 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-3) Running on vds during rerun failed vm: null >2012-11-08 08:52:30,130 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-3) vm b5 running in db and not running in vds - add to rerun treatment. vds rhs-gp-srv11.lab.eng.blr.redhat.com >2012-11-08 08:52:30,133 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-3) Rerun vm 3d766d10-58ab-4306-b5c4-7d109c7b6c46. Called from vds rhs-gp-srv11.lab.eng.blr.redhat.com >2012-11-08 08:52:30,156 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsDynamicDataVDSCommand] (pool-4-thread-49) START, UpdateVdsDynamicDataVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vdsDynamic=org.ovirt.engine.core.common.businessentities.VdsDynamic@a0983f39), log id: 79e46992 >2012-11-08 08:52:30,158 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsDynamicDataVDSCommand] (pool-4-thread-49) FINISH, UpdateVdsDynamicDataVDSCommand, log id: 79e46992 >2012-11-08 08:52:30,175 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (pool-4-thread-49) START, IsValidVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6fc7114e >2012-11-08 08:52:30,175 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (pool-4-thread-49) FINISH, IsValidVDSCommand, return: true, log id: 6fc7114e >2012-11-08 08:52:30,191 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (pool-4-thread-49) START, IsVmDuringInitiatingVDSCommand( vmId = 3d766d10-58ab-4306-b5c4-7d109c7b6c46), log id: 124d13d2 >2012-11-08 08:52:30,191 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (pool-4-thread-49) FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 124d13d2 >2012-11-08 08:52:30,215 INFO [org.ovirt.engine.core.bll.VdsSelector] (pool-4-thread-49) VDS rhs-gp-srv11.lab.eng.blr.redhat.com 96739be4-2990-11e2-8577-52540035ed48 is the same host the VM is currently running on VDS rhs-gp-srv12.lab.eng.blr.redhat.com 62704566-29ab-11e2-82bf-52540035ed48 is not in up status or belongs to the VM's cluster VDS rhs-gp-srv4.lab.eng.blr.redhat.com d2bd32d4-2992-11e2-ab59-52540035ed48 is not in up status or belongs to the VM's cluster VDS rhs-arch-srv2.lab.eng.blr.redhat.com cf017816-29a2-11e2-949e-52540035ed48 is not in up status or belongs to the VM's cluster >2012-11-08 08:52:30,215 WARN [org.ovirt.engine.core.bll.RunVmOnceCommand] (pool-4-thread-49) CanDoAction of action RunVmOnce failed. Reasons:VAR__ACTION__RUN,VAR__TYPE__VM,$VmName b5,VAR__ACTION__RUN,VAR__TYPE__VM,ACTION_TYPE_FAILED_VDS_VM_CLUSTER >2012-11-08 08:52:34,917 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-79) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='(~vdsm~, ~vdsm-cli~)' message='vdsm-4.9.6-41.0.el6_3.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:52:37,637 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-79) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='libjpeg' message='libjpeg-6b-46.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:52:40,115 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-79) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='spice-server' message='spice-server-0.10.1-10.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:52:42,515 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-79) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='pixman' message='pixman-0.18.4-1.el6_0.1.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:52:44,907 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-79) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='seabios' message='seabios-0.6.1.2-19.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:52:47,387 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-79) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='fence-agents' message='fence-agents-3.1.5-17.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:52:49,837 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-79) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='libselinux-python' message='libselinux-python-2.0.94-5.3.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:52:52,238 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-79) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='sanlock' message='sanlock-2.3-4.el6_3.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:52:54,710 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-79) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='sanlock-python' message='sanlock-python-2.3-4.el6_3.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:52:57,174 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-79) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='qemu-img' message='package qemu-img is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:53:09,294 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-79) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='qemu-kvm-tools' message='package qemu-kvm-tools is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:53:11,701 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-79) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='qemu-kvm' message='package qemu-kvm is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:53:11,712 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-79) Installation of 10.70.36.16. Received message: <BSTRAP component='CreateConf' status='OK' message='Basic configuration set'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:53:11,717 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-79) Installation of 10.70.36.16. Received message: <BSTRAP component='CheckLocalHostname' status='OK' message='Local hostname is correct.'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:53:18,020 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-1) [c3b2c8] START, IsValidVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 55b53bed >2012-11-08 08:53:18,021 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-1) [c3b2c8] FINISH, IsValidVDSCommand, return: true, log id: 55b53bed >2012-11-08 08:53:18,032 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp-/127.0.0.1:8702-1) [c3b2c8] START, IsVmDuringInitiatingVDSCommand( vmId = 3d766d10-58ab-4306-b5c4-7d109c7b6c46), log id: 4ba4d755 >2012-11-08 08:53:18,032 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp-/127.0.0.1:8702-1) [c3b2c8] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 4ba4d755 >2012-11-08 08:53:18,068 INFO [org.ovirt.engine.core.bll.RunVmOnceCommand] (ajp-/127.0.0.1:8702-1) [c3b2c8] Running command: RunVmOnceCommand internal: false. Entities affected : ID: 3d766d10-58ab-4306-b5c4-7d109c7b6c46 Type: VM >2012-11-08 08:53:18,120 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsoPrefixVDSCommand] (ajp-/127.0.0.1:8702-1) [c3b2c8] START, IsoPrefixVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 5a703798 >2012-11-08 08:53:18,120 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsoPrefixVDSCommand] (ajp-/127.0.0.1:8702-1) [c3b2c8] FINISH, IsoPrefixVDSCommand, return: /rhev/data-center/3a17dcac-a6e3-4563-a4b0-f27d89b68dd6/7a268ac7-857a-4c53-966d-247deb509161/images/11111111-1111-1111-1111-111111111111, log id: 5a703798 >2012-11-08 08:53:18,138 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-1) [c3b2c8] START, CreateVmVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=3d766d10-58ab-4306-b5c4-7d109c7b6c46, vm=org.ovirt.engine.core.common.businessentities.VM@5900a156), log id: 7d520a8a >2012-11-08 08:53:18,155 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8702-1) [c3b2c8] START, CreateVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=3d766d10-58ab-4306-b5c4-7d109c7b6c46, vm=org.ovirt.engine.core.common.businessentities.VM@5900a156), log id: 6ffe80a8 >2012-11-08 08:53:18,216 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8702-1) [c3b2c8] org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand spiceSslCipherSuite=DEFAULT,memSize=512,kvmEnable=true,smp=1,emulatedMachine=rhel6.3.0,vmType=kvm,keyboardLayout=en-us,nice=0,display=qxl,smpCoresPerSocket=1,spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard,timeOffset=-43200,transparentHugePages=true,vmId=3d766d10-58ab-4306-b5c4-7d109c7b6c46,devices=[Ljava.util.Map;@59640932,acpiEnable=true,vmName=b5,cpuType=Nehalem,custom={} >2012-11-08 08:53:18,216 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8702-1) [c3b2c8] FINISH, CreateVDSCommand, log id: 6ffe80a8 >2012-11-08 08:53:18,216 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-1) [c3b2c8] IncreasePendingVms::CreateVmIncreasing vds rhs-gp-srv11.lab.eng.blr.redhat.com pending vcpu count, now 1. Vm: b5 >2012-11-08 08:53:18,218 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-1) [c3b2c8] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 7d520a8a >2012-11-08 08:53:19,606 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-79) Installation of 10.70.36.16. Received message: <BSTRAP component='SetNetworking' status='OK' message='Created management bridge.'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:53:19,639 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (QuartzScheduler_Worker-21) START, DestroyVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=3d766d10-58ab-4306-b5c4-7d109c7b6c46, force=false, secondsToWait=0, gracefully=false), log id: 2140f807 >2012-11-08 08:53:19,692 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (QuartzScheduler_Worker-21) FINISH, DestroyVDSCommand, log id: 2140f807 >2012-11-08 08:53:19,705 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-79) Installation of 10.70.36.16. Received message: <BSTRAP component='SetSSHAccess' status='OK' message='SUCCESS'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:53:19,708 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-21) Running on vds during rerun failed vm: null >2012-11-08 08:53:19,709 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-21) vm b5 running in db and not running in vds - add to rerun treatment. vds rhs-gp-srv11.lab.eng.blr.redhat.com >2012-11-08 08:53:19,712 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-21) Rerun vm 3d766d10-58ab-4306-b5c4-7d109c7b6c46. Called from vds rhs-gp-srv11.lab.eng.blr.redhat.com >2012-11-08 08:53:19,732 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsDynamicDataVDSCommand] (pool-4-thread-49) START, UpdateVdsDynamicDataVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vdsDynamic=org.ovirt.engine.core.common.businessentities.VdsDynamic@e0159d09), log id: 20905fd4 >2012-11-08 08:53:19,733 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsDynamicDataVDSCommand] (pool-4-thread-49) FINISH, UpdateVdsDynamicDataVDSCommand, log id: 20905fd4 >2012-11-08 08:53:19,750 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (pool-4-thread-49) START, IsValidVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: b351a38 >2012-11-08 08:53:19,751 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (pool-4-thread-49) FINISH, IsValidVDSCommand, return: true, log id: b351a38 >2012-11-08 08:53:19,770 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (pool-4-thread-49) START, IsVmDuringInitiatingVDSCommand( vmId = 3d766d10-58ab-4306-b5c4-7d109c7b6c46), log id: 1c8a668b >2012-11-08 08:53:19,770 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (pool-4-thread-49) FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 1c8a668b >2012-11-08 08:53:19,799 INFO [org.ovirt.engine.core.bll.VdsSelector] (pool-4-thread-49) VDS rhs-gp-srv12.lab.eng.blr.redhat.com 62704566-29ab-11e2-82bf-52540035ed48 is not in up status or belongs to the VM's cluster VDS rhs-gp-srv11.lab.eng.blr.redhat.com 96739be4-2990-11e2-8577-52540035ed48 is the same host the VM is currently running on VDS rhs-gp-srv4.lab.eng.blr.redhat.com d2bd32d4-2992-11e2-ab59-52540035ed48 is not in up status or belongs to the VM's cluster VDS rhs-arch-srv2.lab.eng.blr.redhat.com cf017816-29a2-11e2-949e-52540035ed48 is not in up status or belongs to the VM's cluster >2012-11-08 08:53:19,799 WARN [org.ovirt.engine.core.bll.RunVmOnceCommand] (pool-4-thread-49) CanDoAction of action RunVmOnce failed. Reasons:VAR__ACTION__RUN,VAR__TYPE__VM,$VmName b5,VAR__ACTION__RUN,VAR__TYPE__VM,ACTION_TYPE_FAILED_VDS_VM_CLUSTER >2012-11-08 08:53:19,981 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-79) Installation of 10.70.36.16. Received message: <BSTRAP component='VerifyServices' status='OK' message='Needed services set'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:53:23,669 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-79) Installation of 10.70.36.16. Received message: <BSTRAP component='Encryption setup' status='OK' message='Ended successfully'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 08:53:23,674 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-79) Installation of 10.70.36.16. Received message: <BSTRAP component='RHEV_INSTALL' status='OK'/>. Stage completed. (Stage: Running first installation script on Host) >2012-11-08 08:53:23,706 ERROR [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-45) [46a65120] Installation of 10.70.36.16. Error: Repository JBOSS is listed more than once in the configuration >Repository qa-latest-5 is listed more than once in the configuration >Repository rhel-source is listed more than once in the configuration >Repository rhel-source-beta is listed more than once in the configuration >Repository rhs-client is listed more than once in the configuration >Repository JBOSS is listed more than once in the configuration >Repository qa-latest-5 is listed more than once in the configuration >Repository rhel-source is listed more than once in the configuration >Repository rhel-source-beta is listed more than once in the configuration >Repository rhs-client is listed more than once in the configuration >. (Stage: Downloading certificate request from Host) >2012-11-08 08:53:23,706 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-45) [46a65120] VDS message: Repository JBOSS is listed more than once in the configuration >2012-11-08 08:53:23,706 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-45) [46a65120] VDS message: Repository qa-latest-5 is listed more than once in the configuration >2012-11-08 08:53:23,706 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-45) [46a65120] VDS message: Repository rhel-source is listed more than once in the configuration >2012-11-08 08:53:23,706 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-45) [46a65120] VDS message: Repository rhel-source-beta is listed more than once in the configuration >2012-11-08 08:53:23,706 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-45) [46a65120] VDS message: Repository rhs-client is listed more than once in the configuration >2012-11-08 08:53:23,706 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-45) [46a65120] VDS message: Repository JBOSS is listed more than once in the configuration >2012-11-08 08:53:23,706 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-45) [46a65120] VDS message: Repository qa-latest-5 is listed more than once in the configuration >2012-11-08 08:53:23,706 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-45) [46a65120] VDS message: Repository rhel-source is listed more than once in the configuration >2012-11-08 08:53:23,706 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-45) [46a65120] VDS message: Repository rhel-source-beta is listed more than once in the configuration >2012-11-08 08:53:23,706 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-45) [46a65120] VDS message: Repository rhs-client is listed more than once in the configuration >2012-11-08 08:53:23,706 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-45) [46a65120] Script ended, result is {1} >2012-11-08 08:53:23,707 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-45) [46a65120] Installation of 10.70.36.16. Executing installation stage. (Stage: Downloading certificate request from Host) >2012-11-08 08:53:23,707 INFO [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-4-thread-45) SSH receive 10.70.36.16:'/tmp/cert_0cb47953-de57-4054-b6f9-e09a65dd4fb9.req' '/etc/pki/ovirt-engine/requests/cert_0cb47953-de57-4054-b6f9-e09a65dd4fb9.req') >2012-11-08 08:53:23,779 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-45) [46a65120] Installation of 10.70.36.16. successfully done sftp operation ( Stage: Downloading certificate request from Host) >2012-11-08 08:53:23,779 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-45) [46a65120] DownloadCertificateRequest ended:true >2012-11-08 08:53:23,779 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-45) [46a65120] Installation of 10.70.36.16. Executing installation stage. (Stage: Sign certificate request and generate certificate) >2012-11-08 08:53:24,790 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-45) [46a65120] SignCertificateRequest ended:true >2012-11-08 08:53:24,790 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-45) [46a65120] Installation of 10.70.36.16. Executing installation stage. (Stage: Upload signed sertificate to Host) >2012-11-08 08:53:24,811 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-45) [46a65120] Installation of 10.70.36.16. successfully done sftp operation ( Stage: Upload signed sertificate to Host) >2012-11-08 08:53:24,812 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-45) [46a65120] UploadSignedCertificate ended:true >2012-11-08 08:53:24,812 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-45) [46a65120] Installation of 10.70.36.16. Executing installation stage. (Stage: Upload Cerficate Autority to Host) >2012-11-08 08:53:24,865 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-45) [46a65120] Installation of 10.70.36.16. successfully done sftp operation ( Stage: Upload Cerficate Autority to Host) >2012-11-08 08:53:24,865 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-45) [46a65120] Installation of 10.70.36.16. Executing installation stage. (Stage: Running second installation script on Host) >2012-11-08 08:53:24,865 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-45) [46a65120] Installation of 10.70.36.16. Sending SSH Command umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar -C "${MYTMP}" --no-same-permissions -o -x && "${MYTMP}"/setup -c 'ssl=true;management_port=54321' -O 'lab.eng.blr.redhat.com' -t 2012-11-08T13:51:24 -S /tmp/ovirt-id_rsa_0cb47953-de57-4054-b6f9-e09a65dd4fb9 -p 80 -b -B rhevm http://guido.lab.eng.blr.redhat.com:80/Components/vds/ http://guido.lab.eng.blr.redhat.com:80/Components/vds/ 10.70.36.16 0cb47953-de57-4054-b6f9-e09a65dd4fb9 True < /var/cache/ovirt-engine/vdsm-bootstrap-2.tar. (Stage: Running second installation script on Host) >2012-11-08 08:53:24,866 INFO [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-4-thread-45) SSH execute 10.70.36.16 'umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar -C "${MYTMP}" --no-same-permissions -o -x && "${MYTMP}"/setup -c 'ssl=true;management_port=54321' -O 'lab.eng.blr.redhat.com' -t 2012-11-08T13:51:24 -S /tmp/ovirt-id_rsa_0cb47953-de57-4054-b6f9-e09a65dd4fb9 -p 80 -b -B rhevm http://guido.lab.eng.blr.redhat.com:80/Components/vds/ http://guido.lab.eng.blr.redhat.com:80/Components/vds/ 10.70.36.16 0cb47953-de57-4054-b6f9-e09a65dd4fb9 True' >2012-11-08 08:53:24,956 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-79) Installation of 10.70.36.16. Received message: <BSTRAP component='INSTALLER' status='OK' message='Test platform succeeded'/>. FYI. (Stage: Running second installation script on Host) >2012-11-08 08:53:25,304 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-79) Installation of 10.70.36.16. Received message: <BSTRAP component='instCert' status='OK'/>. FYI. (Stage: Running second installation script on Host) >2012-11-08 08:53:25,308 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-79) Installation of 10.70.36.16. Received message: <BSTRAP component='CoreDump' status='OK'/>. FYI. (Stage: Running second installation script on Host) >2012-11-08 08:53:25,311 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-79) Installation of 10.70.36.16. Received message: <BSTRAP component='cleanAll' status='OK'/>. FYI. (Stage: Running second installation script on Host) >2012-11-08 08:53:25,314 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-79) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS Configuration' status='OK'/>. FYI. (Stage: Running second installation script on Host) >2012-11-08 08:53:25,317 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-79) Installation of 10.70.36.16. Received message: <BSTRAP component='RHEV_INSTALL' status='OK'/>. Stage completed. (Stage: Running second installation script on Host) >2012-11-08 08:53:25,325 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-79) Installation of 10.70.36.16. Received message: <BSTRAP component='Reboot' status='OK' message='Rebooting machine' />. FYI. (Stage: Host installation complete) >2012-11-08 08:53:25,335 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-45) [46a65120] Script ended, result is {1} >2012-11-08 08:53:25,337 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-4-thread-45) [46a65120] After Installation pool-4-thread-45 >2012-11-08 08:53:25,338 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-45) [46a65120] START, SetVdsStatusVDSCommand(HostName = rhs-gp-srv12.lab.eng.blr.redhat.com, HostId = 62704566-29ab-11e2-82bf-52540035ed48, status=Reboot, nonOperationalReason=NONE), log id: 18659a72 >2012-11-08 08:53:25,347 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-45) [46a65120] FINISH, SetVdsStatusVDSCommand, log id: 18659a72 >2012-11-08 08:53:25,348 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-4-thread-10) [46a65120] Waiting 300 seconds, for server to finish reboot process. >2012-11-08 08:54:32,349 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-7) [25c72a82] START, IsValidVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 72061a4f >2012-11-08 08:54:32,353 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-7) [25c72a82] FINISH, IsValidVDSCommand, return: true, log id: 72061a4f >2012-11-08 08:54:32,366 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp-/127.0.0.1:8702-7) [25c72a82] START, IsVmDuringInitiatingVDSCommand( vmId = 3d766d10-58ab-4306-b5c4-7d109c7b6c46), log id: 122f9474 >2012-11-08 08:54:32,366 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp-/127.0.0.1:8702-7) [25c72a82] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 122f9474 >2012-11-08 08:54:32,405 INFO [org.ovirt.engine.core.bll.RunVmOnceCommand] (ajp-/127.0.0.1:8702-7) [25c72a82] Running command: RunVmOnceCommand internal: false. Entities affected : ID: 3d766d10-58ab-4306-b5c4-7d109c7b6c46 Type: VM >2012-11-08 08:54:32,454 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsoPrefixVDSCommand] (ajp-/127.0.0.1:8702-7) [25c72a82] START, IsoPrefixVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 62e5ef37 >2012-11-08 08:54:32,454 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsoPrefixVDSCommand] (ajp-/127.0.0.1:8702-7) [25c72a82] FINISH, IsoPrefixVDSCommand, return: /rhev/data-center/3a17dcac-a6e3-4563-a4b0-f27d89b68dd6/7a268ac7-857a-4c53-966d-247deb509161/images/11111111-1111-1111-1111-111111111111, log id: 62e5ef37 >2012-11-08 08:54:32,472 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-7) [25c72a82] START, CreateVmVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=3d766d10-58ab-4306-b5c4-7d109c7b6c46, vm=org.ovirt.engine.core.common.businessentities.VM@30fb5bd2), log id: 65bc369a >2012-11-08 08:54:32,492 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8702-7) [25c72a82] START, CreateVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=3d766d10-58ab-4306-b5c4-7d109c7b6c46, vm=org.ovirt.engine.core.common.businessentities.VM@30fb5bd2), log id: 475a593a >2012-11-08 08:54:32,542 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8702-7) [25c72a82] org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand spiceSslCipherSuite=DEFAULT,memSize=512,kvmEnable=true,smp=1,emulatedMachine=rhel6.3.0,vmType=kvm,keyboardLayout=en-us,nice=0,display=qxl,smpCoresPerSocket=1,spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard,timeOffset=-43200,transparentHugePages=true,vmId=3d766d10-58ab-4306-b5c4-7d109c7b6c46,devices=[Ljava.util.Map;@6e75deac,acpiEnable=true,vmName=b5,cpuType=Nehalem,custom={} >2012-11-08 08:54:32,542 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8702-7) [25c72a82] FINISH, CreateVDSCommand, log id: 475a593a >2012-11-08 08:54:32,545 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-7) [25c72a82] IncreasePendingVms::CreateVmIncreasing vds rhs-gp-srv11.lab.eng.blr.redhat.com pending vcpu count, now 1. Vm: b5 >2012-11-08 08:54:32,547 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-7) [25c72a82] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 65bc369a >2012-11-08 08:54:33,959 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (QuartzScheduler_Worker-99) START, DestroyVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=3d766d10-58ab-4306-b5c4-7d109c7b6c46, force=false, secondsToWait=0, gracefully=false), log id: 730e776d >2012-11-08 08:54:34,022 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (QuartzScheduler_Worker-99) FINISH, DestroyVDSCommand, log id: 730e776d >2012-11-08 08:54:34,042 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-99) Running on vds during rerun failed vm: null >2012-11-08 08:54:34,043 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-99) vm b5 running in db and not running in vds - add to rerun treatment. vds rhs-gp-srv11.lab.eng.blr.redhat.com >2012-11-08 08:54:34,047 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-99) Rerun vm 3d766d10-58ab-4306-b5c4-7d109c7b6c46. Called from vds rhs-gp-srv11.lab.eng.blr.redhat.com >2012-11-08 08:54:34,074 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsDynamicDataVDSCommand] (pool-4-thread-45) START, UpdateVdsDynamicDataVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vdsDynamic=org.ovirt.engine.core.common.businessentities.VdsDynamic@a2bad647), log id: 42a9f7d5 >2012-11-08 08:54:34,076 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsDynamicDataVDSCommand] (pool-4-thread-45) FINISH, UpdateVdsDynamicDataVDSCommand, log id: 42a9f7d5 >2012-11-08 08:54:34,091 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (pool-4-thread-45) START, IsValidVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 2842bf35 >2012-11-08 08:54:34,092 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (pool-4-thread-45) FINISH, IsValidVDSCommand, return: true, log id: 2842bf35 >2012-11-08 08:54:34,102 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (pool-4-thread-45) START, IsVmDuringInitiatingVDSCommand( vmId = 3d766d10-58ab-4306-b5c4-7d109c7b6c46), log id: 11b578f2 >2012-11-08 08:54:34,102 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (pool-4-thread-45) FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 11b578f2 >2012-11-08 08:54:34,126 INFO [org.ovirt.engine.core.bll.VdsSelector] (pool-4-thread-45) VDS rhs-gp-srv11.lab.eng.blr.redhat.com 96739be4-2990-11e2-8577-52540035ed48 is the same host the VM is currently running on VDS rhs-gp-srv12.lab.eng.blr.redhat.com 62704566-29ab-11e2-82bf-52540035ed48 is not in up status or belongs to the VM's cluster VDS rhs-gp-srv4.lab.eng.blr.redhat.com d2bd32d4-2992-11e2-ab59-52540035ed48 is not in up status or belongs to the VM's cluster VDS rhs-arch-srv2.lab.eng.blr.redhat.com cf017816-29a2-11e2-949e-52540035ed48 is not in up status or belongs to the VM's cluster >2012-11-08 08:54:34,126 WARN [org.ovirt.engine.core.bll.RunVmOnceCommand] (pool-4-thread-45) CanDoAction of action RunVmOnce failed. Reasons:VAR__ACTION__RUN,VAR__TYPE__VM,$VmName b5,VAR__ACTION__RUN,VAR__TYPE__VM,ACTION_TYPE_FAILED_VDS_VM_CLUSTER >2012-11-08 08:55:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-74) Autorecovering hosts is disabled, skipping >2012-11-08 08:55:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-74) Autorecovering storage domains is disabled, skipping >2012-11-08 08:58:25,350 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-10) [46a65120] START, SetVdsStatusVDSCommand(HostName = rhs-gp-srv12.lab.eng.blr.redhat.com, HostId = 62704566-29ab-11e2-82bf-52540035ed48, status=NonResponsive, nonOperationalReason=NONE), log id: 12c8e07 >2012-11-08 08:58:25,355 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-10) [46a65120] FINISH, SetVdsStatusVDSCommand, log id: 12c8e07 >2012-11-08 08:58:26,634 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-71) [3e904d2f] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 08:58:28,646 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-73) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 08:58:30,654 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-75) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 08:58:32,664 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-87) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 08:58:34,673 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-94) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 08:58:36,685 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-89) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 08:58:38,713 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-1) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 08:58:40,722 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-17) [36467d8f] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 08:58:42,732 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-12) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 08:58:44,742 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-18) [7ab5f38b] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 08:58:46,752 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-13) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 08:58:48,764 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-21) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 08:58:50,775 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-26) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 08:58:52,787 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-36) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 08:58:54,798 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-33) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 08:58:56,808 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-38) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 08:58:58,820 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-53) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 08:59:00,831 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-61) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 08:59:02,844 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-68) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 08:59:04,854 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-65) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 08:59:06,862 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-35) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 08:59:08,872 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-85) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 08:59:10,881 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-79) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 08:59:12,891 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-88) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 08:59:14,900 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-98) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 08:59:16,911 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-91) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 08:59:18,922 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-5) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 08:59:19,589 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-5) [35c401e7] START, IsValidVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 49b8dfd3 >2012-11-08 08:59:19,590 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-5) [35c401e7] FINISH, IsValidVDSCommand, return: true, log id: 49b8dfd3 >2012-11-08 08:59:19,601 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp-/127.0.0.1:8702-5) [35c401e7] START, IsVmDuringInitiatingVDSCommand( vmId = 3d766d10-58ab-4306-b5c4-7d109c7b6c46), log id: 3bc0f1c0 >2012-11-08 08:59:19,601 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp-/127.0.0.1:8702-5) [35c401e7] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 3bc0f1c0 >2012-11-08 08:59:19,633 INFO [org.ovirt.engine.core.bll.RunVmOnceCommand] (ajp-/127.0.0.1:8702-5) [35c401e7] Running command: RunVmOnceCommand internal: false. Entities affected : ID: 3d766d10-58ab-4306-b5c4-7d109c7b6c46 Type: VM >2012-11-08 08:59:19,681 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsoPrefixVDSCommand] (ajp-/127.0.0.1:8702-5) [35c401e7] START, IsoPrefixVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 42ba607f >2012-11-08 08:59:19,681 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsoPrefixVDSCommand] (ajp-/127.0.0.1:8702-5) [35c401e7] FINISH, IsoPrefixVDSCommand, return: /rhev/data-center/3a17dcac-a6e3-4563-a4b0-f27d89b68dd6/7a268ac7-857a-4c53-966d-247deb509161/images/11111111-1111-1111-1111-111111111111, log id: 42ba607f >2012-11-08 08:59:19,699 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-5) [35c401e7] START, CreateVmVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=3d766d10-58ab-4306-b5c4-7d109c7b6c46, vm=org.ovirt.engine.core.common.businessentities.VM@757bb7c4), log id: 5a155412 >2012-11-08 08:59:19,711 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8702-5) [35c401e7] START, CreateVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=3d766d10-58ab-4306-b5c4-7d109c7b6c46, vm=org.ovirt.engine.core.common.businessentities.VM@757bb7c4), log id: 874e543 >2012-11-08 08:59:19,762 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8702-5) [35c401e7] org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand spiceSslCipherSuite=DEFAULT,memSize=512,kvmEnable=true,smp=1,emulatedMachine=rhel6.3.0,vmType=kvm,keyboardLayout=en-us,nice=0,display=qxl,smpCoresPerSocket=1,spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard,timeOffset=-43200,transparentHugePages=true,vmId=3d766d10-58ab-4306-b5c4-7d109c7b6c46,devices=[Ljava.util.Map;@7eba1e1f,acpiEnable=true,vmName=b5,cpuType=Nehalem,custom={} >2012-11-08 08:59:19,762 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8702-5) [35c401e7] FINISH, CreateVDSCommand, log id: 874e543 >2012-11-08 08:59:19,763 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-5) [35c401e7] IncreasePendingVms::CreateVmIncreasing vds rhs-gp-srv11.lab.eng.blr.redhat.com pending vcpu count, now 1. Vm: b5 >2012-11-08 08:59:19,766 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-5) [35c401e7] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 5a155412 >2012-11-08 08:59:20,939 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-17) [36467d8f] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 08:59:22,222 INFO [org.ovirt.engine.core.bll.storage.FenceVdsManualyCommand] (ajp-/127.0.0.1:8702-14) Lock Acquired to object EngineLock [exclusiveLocks= key: 62704566-29ab-11e2-82bf-52540035ed48 value: VDS_FENCE >, sharedLocks= ] >2012-11-08 08:59:22,223 INFO [org.ovirt.engine.core.bll.storage.FenceVdsManualyCommand] (pool-4-thread-10) [86da94b] Running command: FenceVdsManualyCommand internal: false. Entities affected : ID: 62704566-29ab-11e2-82bf-52540035ed48 Type: VDS >2012-11-08 08:59:22,229 INFO [org.ovirt.engine.core.bll.ClearNonResponsiveVdsVmsCommand] (pool-4-thread-10) [27ea62c9] Running command: ClearNonResponsiveVdsVmsCommand internal: true. Entities affected : ID: 62704566-29ab-11e2-82bf-52540035ed48 Type: VDS >2012-11-08 08:59:22,231 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsVMsClearedVDSCommand] (pool-4-thread-10) [27ea62c9] START, UpdateVdsVMsClearedVDSCommand(HostName = rhs-gp-srv12.lab.eng.blr.redhat.com, HostId = 62704566-29ab-11e2-82bf-52540035ed48), log id: d5047cf >2012-11-08 08:59:22,232 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsVMsClearedVDSCommand] (pool-4-thread-10) [27ea62c9] FINISH, UpdateVdsVMsClearedVDSCommand, log id: d5047cf >2012-11-08 08:59:22,238 INFO [org.ovirt.engine.core.bll.storage.FenceVdsManualyCommand] (pool-4-thread-10) [27ea62c9] Lock freed to object EngineLock [exclusiveLocks= key: 62704566-29ab-11e2-82bf-52540035ed48 value: VDS_FENCE >, sharedLocks= ] >2012-11-08 08:59:22,327 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (QuartzScheduler_Worker-7) START, DestroyVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=3d766d10-58ab-4306-b5c4-7d109c7b6c46, force=false, secondsToWait=0, gracefully=false), log id: 2f2b73f8 >2012-11-08 08:59:22,385 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (QuartzScheduler_Worker-7) FINISH, DestroyVDSCommand, log id: 2f2b73f8 >2012-11-08 08:59:22,406 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-7) Running on vds during rerun failed vm: null >2012-11-08 08:59:22,406 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-7) vm b5 running in db and not running in vds - add to rerun treatment. vds rhs-gp-srv11.lab.eng.blr.redhat.com >2012-11-08 08:59:22,409 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-7) Rerun vm 3d766d10-58ab-4306-b5c4-7d109c7b6c46. Called from vds rhs-gp-srv11.lab.eng.blr.redhat.com >2012-11-08 08:59:22,443 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsDynamicDataVDSCommand] (pool-4-thread-49) START, UpdateVdsDynamicDataVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vdsDynamic=org.ovirt.engine.core.common.businessentities.VdsDynamic@65600f85), log id: 750db889 >2012-11-08 08:59:22,445 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsDynamicDataVDSCommand] (pool-4-thread-49) FINISH, UpdateVdsDynamicDataVDSCommand, log id: 750db889 >2012-11-08 08:59:22,469 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (pool-4-thread-49) START, IsValidVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3ccd6f86 >2012-11-08 08:59:22,470 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (pool-4-thread-49) FINISH, IsValidVDSCommand, return: true, log id: 3ccd6f86 >2012-11-08 08:59:22,487 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (pool-4-thread-49) START, IsVmDuringInitiatingVDSCommand( vmId = 3d766d10-58ab-4306-b5c4-7d109c7b6c46), log id: 6464bfb5 >2012-11-08 08:59:22,487 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (pool-4-thread-49) FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 6464bfb5 >2012-11-08 08:59:22,513 INFO [org.ovirt.engine.core.bll.VdsSelector] (pool-4-thread-49) VDS rhs-gp-srv12.lab.eng.blr.redhat.com 62704566-29ab-11e2-82bf-52540035ed48 is not in up status or belongs to the VM's cluster VDS rhs-gp-srv11.lab.eng.blr.redhat.com 96739be4-2990-11e2-8577-52540035ed48 is the same host the VM is currently running on VDS rhs-gp-srv4.lab.eng.blr.redhat.com d2bd32d4-2992-11e2-ab59-52540035ed48 is not in up status or belongs to the VM's cluster VDS rhs-arch-srv2.lab.eng.blr.redhat.com cf017816-29a2-11e2-949e-52540035ed48 is not in up status or belongs to the VM's cluster >2012-11-08 08:59:22,514 WARN [org.ovirt.engine.core.bll.RunVmOnceCommand] (pool-4-thread-49) CanDoAction of action RunVmOnce failed. Reasons:VAR__ACTION__RUN,VAR__TYPE__VM,$VmName b5,VAR__ACTION__RUN,VAR__TYPE__VM,ACTION_TYPE_FAILED_VDS_VM_CLUSTER >2012-11-08 08:59:22,950 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-12) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 08:59:24,961 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-18) [7ab5f38b] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 08:59:26,970 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-23) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 08:59:28,984 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-34) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 08:59:30,923 INFO [org.ovirt.engine.core.bll.MaintananceNumberOfVdssCommand] (pool-4-thread-49) [4ffb42f4] Running command: MaintananceNumberOfVdssCommand internal: false. Entities affected : ID: 62704566-29ab-11e2-82bf-52540035ed48 Type: VDS >2012-11-08 08:59:30,935 INFO [org.ovirt.engine.core.bll.MaintananceVdsCommand] (pool-4-thread-49) [4ffb42f4] Running command: MaintananceVdsCommand internal: true. Entities affected : ID: 62704566-29ab-11e2-82bf-52540035ed48 Type: VDS >2012-11-08 08:59:30,937 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-49) [4ffb42f4] START, SetVdsStatusVDSCommand(HostName = rhs-gp-srv12.lab.eng.blr.redhat.com, HostId = 62704566-29ab-11e2-82bf-52540035ed48, status=Maintenance, nonOperationalReason=NONE), log id: 277308d3 >2012-11-08 08:59:30,939 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-49) [4ffb42f4] FINISH, SetVdsStatusVDSCommand, log id: 277308d3 >2012-11-08 08:59:34,550 INFO [org.ovirt.engine.core.bll.RunVmCommand] (ajp-/127.0.0.1:8702-13) Lock Acquired to object EngineLock [exclusiveLocks= key: 3d766d10-58ab-4306-b5c4-7d109c7b6c46 value: VM >, sharedLocks= ] >2012-11-08 08:59:34,559 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-13) START, IsValidVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: f5812dd >2012-11-08 08:59:34,559 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-13) FINISH, IsValidVDSCommand, return: true, log id: f5812dd >2012-11-08 08:59:34,572 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp-/127.0.0.1:8702-13) START, IsVmDuringInitiatingVDSCommand( vmId = 3d766d10-58ab-4306-b5c4-7d109c7b6c46), log id: 4da7835c >2012-11-08 08:59:34,573 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp-/127.0.0.1:8702-13) FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 4da7835c >2012-11-08 08:59:34,620 INFO [org.ovirt.engine.core.bll.RunVmCommand] (pool-4-thread-49) [643a209e] Running command: RunVmCommand internal: false. Entities affected : ID: 3d766d10-58ab-4306-b5c4-7d109c7b6c46 Type: VM >2012-11-08 08:59:34,677 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsoPrefixVDSCommand] (pool-4-thread-49) [643a209e] START, IsoPrefixVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 1e60710 >2012-11-08 08:59:34,677 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsoPrefixVDSCommand] (pool-4-thread-49) [643a209e] FINISH, IsoPrefixVDSCommand, return: /rhev/data-center/3a17dcac-a6e3-4563-a4b0-f27d89b68dd6/7a268ac7-857a-4c53-966d-247deb509161/images/11111111-1111-1111-1111-111111111111, log id: 1e60710 >2012-11-08 08:59:34,690 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (pool-4-thread-49) [643a209e] START, CreateVmVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=3d766d10-58ab-4306-b5c4-7d109c7b6c46, vm=org.ovirt.engine.core.common.businessentities.VM@3567b2ef), log id: 24d9ac69 >2012-11-08 08:59:34,712 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (pool-4-thread-49) [643a209e] START, CreateVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=3d766d10-58ab-4306-b5c4-7d109c7b6c46, vm=org.ovirt.engine.core.common.businessentities.VM@3567b2ef), log id: 1b0364e6 >2012-11-08 08:59:34,766 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (pool-4-thread-49) [643a209e] org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand spiceSslCipherSuite=DEFAULT,memSize=512,kvmEnable=true,smp=1,emulatedMachine=rhel6.3.0,vmType=kvm,keyboardLayout=en-us,nice=0,display=qxl,smpCoresPerSocket=1,spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard,timeOffset=-43200,transparentHugePages=true,vmId=3d766d10-58ab-4306-b5c4-7d109c7b6c46,devices=[Ljava.util.Map;@34865fa8,acpiEnable=true,vmName=b5,cpuType=Nehalem,custom={} >2012-11-08 08:59:34,766 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (pool-4-thread-49) [643a209e] FINISH, CreateVDSCommand, log id: 1b0364e6 >2012-11-08 08:59:34,766 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (pool-4-thread-49) [643a209e] IncreasePendingVms::CreateVmIncreasing vds rhs-gp-srv11.lab.eng.blr.redhat.com pending vcpu count, now 1. Vm: b5 >2012-11-08 08:59:34,771 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (pool-4-thread-49) [643a209e] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 24d9ac69 >2012-11-08 08:59:34,773 INFO [org.ovirt.engine.core.bll.RunVmCommand] (pool-4-thread-49) [643a209e] Lock freed to object EngineLock [exclusiveLocks= key: 3d766d10-58ab-4306-b5c4-7d109c7b6c46 value: VM >, sharedLocks= ] >2012-11-08 08:59:36,770 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (QuartzScheduler_Worker-49) START, DestroyVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=3d766d10-58ab-4306-b5c4-7d109c7b6c46, force=false, secondsToWait=0, gracefully=false), log id: 13fdb90 >2012-11-08 08:59:36,833 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (QuartzScheduler_Worker-49) FINISH, DestroyVDSCommand, log id: 13fdb90 >2012-11-08 08:59:36,851 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-49) Running on vds during rerun failed vm: null >2012-11-08 08:59:36,852 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-49) vm b5 running in db and not running in vds - add to rerun treatment. vds rhs-gp-srv11.lab.eng.blr.redhat.com >2012-11-08 08:59:36,855 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-49) Rerun vm 3d766d10-58ab-4306-b5c4-7d109c7b6c46. Called from vds rhs-gp-srv11.lab.eng.blr.redhat.com >2012-11-08 08:59:36,881 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsDynamicDataVDSCommand] (pool-4-thread-49) START, UpdateVdsDynamicDataVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vdsDynamic=org.ovirt.engine.core.common.businessentities.VdsDynamic@65600f85), log id: 3e1a560 >2012-11-08 08:59:36,882 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsDynamicDataVDSCommand] (pool-4-thread-49) FINISH, UpdateVdsDynamicDataVDSCommand, log id: 3e1a560 >2012-11-08 08:59:36,895 INFO [org.ovirt.engine.core.bll.RunVmCommand] (pool-4-thread-49) Lock Acquired to object EngineLock [exclusiveLocks= key: 3d766d10-58ab-4306-b5c4-7d109c7b6c46 value: VM >, sharedLocks= ] >2012-11-08 08:59:36,900 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (pool-4-thread-49) START, IsValidVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 20ddb34e >2012-11-08 08:59:36,902 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (pool-4-thread-49) FINISH, IsValidVDSCommand, return: true, log id: 20ddb34e >2012-11-08 08:59:36,917 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (pool-4-thread-49) START, IsVmDuringInitiatingVDSCommand( vmId = 3d766d10-58ab-4306-b5c4-7d109c7b6c46), log id: 1a7f08fc >2012-11-08 08:59:36,918 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (pool-4-thread-49) FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 1a7f08fc >2012-11-08 08:59:36,943 INFO [org.ovirt.engine.core.bll.VdsSelector] (pool-4-thread-49) VDS rhs-gp-srv11.lab.eng.blr.redhat.com 96739be4-2990-11e2-8577-52540035ed48 is the same host the VM is currently running on VDS rhs-gp-srv4.lab.eng.blr.redhat.com d2bd32d4-2992-11e2-ab59-52540035ed48 is not in up status or belongs to the VM's cluster VDS rhs-gp-srv12.lab.eng.blr.redhat.com 62704566-29ab-11e2-82bf-52540035ed48 is not in up status or belongs to the VM's cluster VDS rhs-arch-srv2.lab.eng.blr.redhat.com cf017816-29a2-11e2-949e-52540035ed48 is not in up status or belongs to the VM's cluster >2012-11-08 08:59:36,943 WARN [org.ovirt.engine.core.bll.RunVmCommand] (pool-4-thread-49) CanDoAction of action RunVm failed. Reasons:VAR__ACTION__RUN,VAR__TYPE__VM,$VmName b5,VAR__ACTION__RUN,VAR__TYPE__VM,VAR__ACTION__RUN,VAR__TYPE__VM,ACTION_TYPE_FAILED_VDS_VM_CLUSTER >2012-11-08 08:59:36,943 INFO [org.ovirt.engine.core.bll.RunVmCommand] (pool-4-thread-49) Lock freed to object EngineLock [exclusiveLocks= key: 3d766d10-58ab-4306-b5c4-7d109c7b6c46 value: VM >, sharedLocks= ] >2012-11-08 08:59:46,390 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (ajp-/127.0.0.1:8702-9) Lock Acquired to object EngineLock [exclusiveLocks= key: 62704566-29ab-11e2-82bf-52540035ed48 value: VDS >, sharedLocks= ] >2012-11-08 08:59:46,411 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (pool-4-thread-10) [16be8b94] Running command: ActivateVdsCommand internal: false. Entities affected : ID: 62704566-29ab-11e2-82bf-52540035ed48 Type: VDS >2012-11-08 08:59:46,417 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-10) [16be8b94] START, SetVdsStatusVDSCommand(HostName = rhs-gp-srv12.lab.eng.blr.redhat.com, HostId = 62704566-29ab-11e2-82bf-52540035ed48, status=Unassigned, nonOperationalReason=NONE), log id: 72ab9472 >2012-11-08 08:59:46,419 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-10) [16be8b94] FINISH, SetVdsStatusVDSCommand, log id: 72ab9472 >2012-11-08 08:59:46,424 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (pool-4-thread-10) [16be8b94] START, ActivateVdsVDSCommand(HostName = rhs-gp-srv12.lab.eng.blr.redhat.com, HostId = 62704566-29ab-11e2-82bf-52540035ed48), log id: 75ac85a1 >2012-11-08 08:59:46,427 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (pool-4-thread-10) [16be8b94] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 08:59:46,427 ERROR [org.ovirt.engine.core.vdsbroker.VdsManager] (pool-4-thread-10) [16be8b94] VDS::handleNetworkException Server failed to respond, vds_id = 62704566-29ab-11e2-82bf-52540035ed48, vds_name = rhs-gp-srv12.lab.eng.blr.redhat.com, error = VDSNetworkException: >2012-11-08 08:59:46,439 INFO [org.ovirt.engine.core.bll.VdsEventListener] (pool-4-thread-49) [16be8b94] ResourceManager::vdsNotResponding entered for Host 62704566-29ab-11e2-82bf-52540035ed48, 10.70.36.16 >2012-11-08 08:59:46,444 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (pool-4-thread-10) [16be8b94] ResourceManager::activateVds - failed to get VDS = 62704566-29ab-11e2-82bf-52540035ed48 capabilities with error: VDSNetworkException: . >2012-11-08 08:59:46,444 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (pool-4-thread-10) [16be8b94] ResourceManager::activateVds - failed to activate VDS = 62704566-29ab-11e2-82bf-52540035ed48 >2012-11-08 08:59:46,448 WARN [org.ovirt.engine.core.bll.VdsNotRespondingTreatmentCommand] (pool-4-thread-49) [27e63ec2] CanDoAction of action VdsNotRespondingTreatment failed. Reasons:VDS_FENCING_DISABLED >2012-11-08 08:59:46,452 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (pool-4-thread-10) [e244a4] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 62704566-29ab-11e2-82bf-52540035ed48 Type: VDS >2012-11-08 08:59:46,463 WARN [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (pool-4-thread-10) [1162084f] CanDoAction of action HandleVdsVersion failed. Reasons:VDS_CANNOT_CHECK_VERSION_HOST_NON_RESPONSIVE >2012-11-08 08:59:46,463 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (pool-4-thread-10) [1162084f] FINISH, ActivateVdsVDSCommand, log id: 75ac85a1 >2012-11-08 08:59:46,481 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (pool-4-thread-10) [1162084f] Lock freed to object EngineLock [exclusiveLocks= key: 62704566-29ab-11e2-82bf-52540035ed48 value: VDS >, sharedLocks= ] >2012-11-08 08:59:47,010 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-35) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 08:59:49,021 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-74) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 08:59:51,032 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-79) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 08:59:53,041 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-88) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 08:59:55,051 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-98) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 08:59:57,063 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-96) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 08:59:59,073 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-3) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:00:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-6) Autorecovering hosts is disabled, skipping >2012-11-08 09:00:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-6) Autorecovering storage domains is disabled, skipping >2012-11-08 09:00:01,084 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-7) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:00:03,095 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-16) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:00:05,106 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-25) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:00:07,117 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-27) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:00:09,128 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-30) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:00:11,139 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-9) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:00:13,151 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-32) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:00:15,164 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-45) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:00:15,658 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsoPrefixVDSCommand] (ajp-/127.0.0.1:8702-13) [34419cb3] START, IsoPrefixVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 62134f5e >2012-11-08 09:00:15,658 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsoPrefixVDSCommand] (ajp-/127.0.0.1:8702-13) [34419cb3] FINISH, IsoPrefixVDSCommand, return: /rhev/data-center/3a17dcac-a6e3-4563-a4b0-f27d89b68dd6/7a268ac7-857a-4c53-966d-247deb509161/images/11111111-1111-1111-1111-111111111111, log id: 62134f5e >2012-11-08 09:00:15,686 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-1) [2ec4fd8c] START, IsValidVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 68ec3691 >2012-11-08 09:00:15,687 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-13) [34419cb3] START, IsValidVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 9571938 >2012-11-08 09:00:15,687 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-1) [2ec4fd8c] FINISH, IsValidVDSCommand, return: true, log id: 68ec3691 >2012-11-08 09:00:15,687 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-13) [34419cb3] FINISH, IsValidVDSCommand, return: true, log id: 9571938 >2012-11-08 09:00:15,700 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp-/127.0.0.1:8702-13) [34419cb3] START, IsVmDuringInitiatingVDSCommand( vmId = 3d766d10-58ab-4306-b5c4-7d109c7b6c46), log id: 199bcb65 >2012-11-08 09:00:15,700 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp-/127.0.0.1:8702-13) [34419cb3] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 199bcb65 >2012-11-08 09:00:15,703 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp-/127.0.0.1:8702-1) [2ec4fd8c] START, IsVmDuringInitiatingVDSCommand( vmId = 3d766d10-58ab-4306-b5c4-7d109c7b6c46), log id: 5375da97 >2012-11-08 09:00:15,703 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp-/127.0.0.1:8702-1) [2ec4fd8c] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 5375da97 >2012-11-08 09:00:15,726 INFO [org.ovirt.engine.core.bll.RunVmOnceCommand] (ajp-/127.0.0.1:8702-13) [34419cb3] Running command: RunVmOnceCommand internal: false. Entities affected : ID: 3d766d10-58ab-4306-b5c4-7d109c7b6c46 Type: VM >2012-11-08 09:00:15,731 INFO [org.ovirt.engine.core.bll.RunVmOnceCommand] (ajp-/127.0.0.1:8702-1) [2ec4fd8c] Running command: RunVmOnceCommand internal: false. Entities affected : ID: 3d766d10-58ab-4306-b5c4-7d109c7b6c46 Type: VM >2012-11-08 09:00:15,772 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsoPrefixVDSCommand] (ajp-/127.0.0.1:8702-1) [2ec4fd8c] START, IsoPrefixVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 66771703 >2012-11-08 09:00:15,772 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsoPrefixVDSCommand] (ajp-/127.0.0.1:8702-1) [2ec4fd8c] FINISH, IsoPrefixVDSCommand, return: /rhev/data-center/3a17dcac-a6e3-4563-a4b0-f27d89b68dd6/7a268ac7-857a-4c53-966d-247deb509161/images/11111111-1111-1111-1111-111111111111, log id: 66771703 >2012-11-08 09:00:15,777 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-13) [34419cb3] START, CreateVmVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=3d766d10-58ab-4306-b5c4-7d109c7b6c46, vm=org.ovirt.engine.core.common.businessentities.VM@1416e53d), log id: 54c88f60 >2012-11-08 09:00:15,781 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-1) [2ec4fd8c] START, CreateVmVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=3d766d10-58ab-4306-b5c4-7d109c7b6c46, vm=org.ovirt.engine.core.common.businessentities.VM@2f52ee7a), log id: 1487fa83 >2012-11-08 09:00:15,787 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8702-13) [34419cb3] START, CreateVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=3d766d10-58ab-4306-b5c4-7d109c7b6c46, vm=org.ovirt.engine.core.common.businessentities.VM@1416e53d), log id: 6da6a3db >2012-11-08 09:00:15,835 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8702-13) [34419cb3] org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand spiceSslCipherSuite=DEFAULT,memSize=512,kvmEnable=true,smp=1,emulatedMachine=rhel6.3.0,vmType=kvm,keyboardLayout=en-us,nice=0,display=qxl,smpCoresPerSocket=1,spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard,timeOffset=-43200,transparentHugePages=true,vmId=3d766d10-58ab-4306-b5c4-7d109c7b6c46,devices=[Ljava.util.Map;@ea68b83,acpiEnable=true,vmName=b5,cpuType=Nehalem,custom={} >2012-11-08 09:00:15,835 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8702-13) [34419cb3] FINISH, CreateVDSCommand, log id: 6da6a3db >2012-11-08 09:00:15,836 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-13) [34419cb3] IncreasePendingVms::CreateVmIncreasing vds rhs-gp-srv11.lab.eng.blr.redhat.com pending vcpu count, now 1. Vm: b5 >2012-11-08 09:00:15,837 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-13) [34419cb3] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 54c88f60 >2012-11-08 09:00:15,838 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-1) [2ec4fd8c] Vm Running failed - vm 3d766d10-58ab-4306-b5c4-7d109c7b6c46:b5 already running >2012-11-08 09:00:15,838 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-1) [2ec4fd8c] FINISH, CreateVmVDSCommand, return: Down, log id: 1487fa83 >2012-11-08 09:00:15,840 INFO [org.ovirt.engine.core.bll.RunVmCommand] (ajp-/127.0.0.1:8702-1) [2ec4fd8c] Failed to run desktop b5, rerun >2012-11-08 09:00:15,861 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsDynamicDataVDSCommand] (pool-4-thread-10) [2ec4fd8c] START, UpdateVdsDynamicDataVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vdsDynamic=org.ovirt.engine.core.common.businessentities.VdsDynamic@280548c3), log id: 10dc843a >2012-11-08 09:00:15,862 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsDynamicDataVDSCommand] (pool-4-thread-10) [2ec4fd8c] FINISH, UpdateVdsDynamicDataVDSCommand, log id: 10dc843a >2012-11-08 09:00:15,899 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (pool-4-thread-10) [2ec4fd8c] START, IsValidVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 1fde38e9 >2012-11-08 09:00:15,900 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (pool-4-thread-10) [2ec4fd8c] FINISH, IsValidVDSCommand, return: true, log id: 1fde38e9 >2012-11-08 09:00:15,916 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (pool-4-thread-10) [2ec4fd8c] START, IsVmDuringInitiatingVDSCommand( vmId = 3d766d10-58ab-4306-b5c4-7d109c7b6c46), log id: 4ea7b4a0 >2012-11-08 09:00:15,916 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (pool-4-thread-10) [2ec4fd8c] FINISH, IsVmDuringInitiatingVDSCommand, return: true, log id: 4ea7b4a0 >2012-11-08 09:00:15,917 WARN [org.ovirt.engine.core.bll.RunVmOnceCommand] (pool-4-thread-10) [2ec4fd8c] CanDoAction of action RunVmOnce failed. Reasons:VAR__ACTION__RUN,VAR__TYPE__VM,$VmName b5,VAR__ACTION__RUN,VAR__TYPE__VM,ACTION_TYPE_FAILED_VM_IS_RUNNING >2012-11-08 09:00:17,176 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-48) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:00:18,110 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (QuartzScheduler_Worker-53) START, DestroyVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=3d766d10-58ab-4306-b5c4-7d109c7b6c46, force=false, secondsToWait=0, gracefully=false), log id: 351749e >2012-11-08 09:00:18,170 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (QuartzScheduler_Worker-53) FINISH, DestroyVDSCommand, log id: 351749e >2012-11-08 09:00:18,211 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-53) Running on vds during rerun failed vm: null >2012-11-08 09:00:18,212 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-53) vm b5 running in db and not running in vds - add to rerun treatment. vds rhs-gp-srv11.lab.eng.blr.redhat.com >2012-11-08 09:00:18,223 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-53) Rerun vm 3d766d10-58ab-4306-b5c4-7d109c7b6c46. Called from vds rhs-gp-srv11.lab.eng.blr.redhat.com >2012-11-08 09:00:18,253 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (pool-4-thread-49) START, IsValidVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 798a6a30 >2012-11-08 09:00:18,254 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (pool-4-thread-49) FINISH, IsValidVDSCommand, return: true, log id: 798a6a30 >2012-11-08 09:00:18,272 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (pool-4-thread-49) START, IsVmDuringInitiatingVDSCommand( vmId = 3d766d10-58ab-4306-b5c4-7d109c7b6c46), log id: b98725e >2012-11-08 09:00:18,272 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (pool-4-thread-49) FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: b98725e >2012-11-08 09:00:18,306 INFO [org.ovirt.engine.core.bll.VdsSelector] (pool-4-thread-49) VDS rhs-gp-srv12.lab.eng.blr.redhat.com 62704566-29ab-11e2-82bf-52540035ed48 is not in up status or belongs to the VM's cluster VDS rhs-gp-srv11.lab.eng.blr.redhat.com 96739be4-2990-11e2-8577-52540035ed48 have failed running this VM in the current selection cycle VDS rhs-gp-srv4.lab.eng.blr.redhat.com d2bd32d4-2992-11e2-ab59-52540035ed48 is not in up status or belongs to the VM's cluster VDS rhs-arch-srv2.lab.eng.blr.redhat.com cf017816-29a2-11e2-949e-52540035ed48 is not in up status or belongs to the VM's cluster >2012-11-08 09:00:18,306 WARN [org.ovirt.engine.core.bll.RunVmOnceCommand] (pool-4-thread-49) CanDoAction of action RunVmOnce failed. Reasons:VAR__ACTION__RUN,VAR__TYPE__VM,$VmName b5,VAR__ACTION__RUN,VAR__TYPE__VM,ACTION_TYPE_FAILED_VM_IS_RUNNING,VAR__ACTION__RUN,VAR__TYPE__VM,ACTION_TYPE_FAILED_VDS_VM_CLUSTER >2012-11-08 09:00:19,188 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-54) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:00:21,199 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-56) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:00:23,211 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-46) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:00:25,222 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-64) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:00:27,235 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-85) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:00:29,246 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-82) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:00:31,257 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-88) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:00:33,268 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-98) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:00:35,278 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-91) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:00:37,290 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-95) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:00:39,301 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-3) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:00:41,312 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-2) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:00:43,324 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-24) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:00:45,334 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-11) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:00:47,344 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-15) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:00:49,355 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-28) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:00:51,365 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-41) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:00:52,908 INFO [org.ovirt.engine.core.bll.RemoveVdsCommand] (ajp-/127.0.0.1:8702-13) Lock Acquired to object EngineLock [exclusiveLocks= key: 62704566-29ab-11e2-82bf-52540035ed48 value: VDS >, sharedLocks= ] >2012-11-08 09:00:52,916 INFO [org.ovirt.engine.core.bll.RemoveVdsCommand] (pool-4-thread-49) [8bccad1] Running command: RemoveVdsCommand internal: false. Entities affected : ID: 62704566-29ab-11e2-82bf-52540035ed48 Type: VDS >2012-11-08 09:00:52,927 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (pool-4-thread-49) [8bccad1] START, RemoveVdsVDSCommand( HostId = 62704566-29ab-11e2-82bf-52540035ed48), log id: 43670037 >2012-11-08 09:00:52,927 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (pool-4-thread-49) [8bccad1] vdsManager::disposing >2012-11-08 09:00:52,927 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (pool-4-thread-49) [8bccad1] FINISH, RemoveVdsVDSCommand, log id: 43670037 >2012-11-08 09:00:52,928 INFO [org.ovirt.engine.core.bll.RemoveVdsCommand] (pool-4-thread-49) [8bccad1] Lock freed to object EngineLock [exclusiveLocks= key: 62704566-29ab-11e2-82bf-52540035ed48 value: VDS >, sharedLocks= ] >2012-11-08 09:00:59,437 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsoPrefixVDSCommand] (ajp-/127.0.0.1:8702-7) [4178af72] START, IsoPrefixVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: d224a03 >2012-11-08 09:00:59,437 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsoPrefixVDSCommand] (ajp-/127.0.0.1:8702-7) [4178af72] FINISH, IsoPrefixVDSCommand, return: /rhev/data-center/3a17dcac-a6e3-4563-a4b0-f27d89b68dd6/7a268ac7-857a-4c53-966d-247deb509161/images/11111111-1111-1111-1111-111111111111, log id: d224a03 >2012-11-08 09:00:59,457 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-7) [4178af72] START, IsValidVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6166d27c >2012-11-08 09:00:59,458 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-7) [4178af72] FINISH, IsValidVDSCommand, return: true, log id: 6166d27c >2012-11-08 09:00:59,472 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp-/127.0.0.1:8702-7) [4178af72] START, IsVmDuringInitiatingVDSCommand( vmId = 3d766d10-58ab-4306-b5c4-7d109c7b6c46), log id: 3d1e8ebb >2012-11-08 09:00:59,472 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp-/127.0.0.1:8702-7) [4178af72] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 3d1e8ebb >2012-11-08 09:00:59,500 INFO [org.ovirt.engine.core.bll.RunVmOnceCommand] (ajp-/127.0.0.1:8702-7) [4178af72] Running command: RunVmOnceCommand internal: false. Entities affected : ID: 3d766d10-58ab-4306-b5c4-7d109c7b6c46 Type: VM >2012-11-08 09:00:59,568 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-7) [4178af72] START, CreateVmVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=3d766d10-58ab-4306-b5c4-7d109c7b6c46, vm=org.ovirt.engine.core.common.businessentities.VM@4973c251), log id: 1740a3ba >2012-11-08 09:00:59,580 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8702-7) [4178af72] START, CreateVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=3d766d10-58ab-4306-b5c4-7d109c7b6c46, vm=org.ovirt.engine.core.common.businessentities.VM@4973c251), log id: 4e6673cd >2012-11-08 09:00:59,633 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8702-7) [4178af72] org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand spiceSslCipherSuite=DEFAULT,memSize=512,kvmEnable=true,smp=1,emulatedMachine=rhel6.3.0,vmType=kvm,keyboardLayout=en-us,nice=0,display=qxl,smpCoresPerSocket=1,spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard,timeOffset=-43200,transparentHugePages=true,vmId=3d766d10-58ab-4306-b5c4-7d109c7b6c46,devices=[Ljava.util.Map;@706fcc3e,acpiEnable=true,vmName=b5,cpuType=Nehalem,custom={} >2012-11-08 09:00:59,633 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (ajp-/127.0.0.1:8702-7) [4178af72] FINISH, CreateVDSCommand, log id: 4e6673cd >2012-11-08 09:00:59,634 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-7) [4178af72] IncreasePendingVms::CreateVmIncreasing vds rhs-gp-srv11.lab.eng.blr.redhat.com pending vcpu count, now 1. Vm: b5 >2012-11-08 09:00:59,636 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (ajp-/127.0.0.1:8702-7) [4178af72] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 1740a3ba >2012-11-08 09:01:01,557 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-57) VM b5 3d766d10-58ab-4306-b5c4-7d109c7b6c46 moved from WaitForLaunch --> PoweringUp >2012-11-08 09:01:01,562 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (QuartzScheduler_Worker-57) START, FullListVdsCommand( HostId = 96739be4-2990-11e2-8577-52540035ed48, vds=null, vmIds=[3d766d10-58ab-4306-b5c4-7d109c7b6c46]), log id: 4d42de5a >2012-11-08 09:01:01,583 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (QuartzScheduler_Worker-57) FINISH, FullListVdsCommand, return: [Lorg.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcStruct;@61140e56, log id: 4d42de5a >2012-11-08 09:01:36,784 INFO [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (ajp-/127.0.0.1:8702-5) SSH execute 10.70.36.19 'IDFILE=/etc/vdsm/vdsm.id; if [ -r "${IDFILE}" ]; then cat "${IDFILE}"; else UUID="$(dmidecode -s system-uuid 2> /dev/null | sed -e 's/.*Not.*//' )"; if [ -z "${UUID}" ]; then UUID="$(uuidgen 2> /dev/null)" && mkdir -p "$(dirname "${IDFILE}")" && echo "${UUID}" > "${IDFILE}" && chmod 0644 "${IDFILE}"; fi; [ -n "${UUID}" ] && echo "${UUID}"; fi' >2012-11-08 09:01:36,932 INFO [org.ovirt.engine.core.bll.AddVdsCommand] (ajp-/127.0.0.1:8702-5) [1cc5ace1] Running command: AddVdsCommand internal: false. Entities affected : ID: c04832b4-2976-11e2-b6fb-52540035ed48 Type: VdsGroups >2012-11-08 09:01:36,951 INFO [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp-/127.0.0.1:8702-5) [652fe0c6] Running command: AddVdsSpmIdCommand internal: true. Entities affected : ID: cfa543b0-29ac-11e2-b6f4-52540035ed48 Type: VDS >2012-11-08 09:01:36,954 INFO [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp-/127.0.0.1:8702-5) [652fe0c6] Lock freed to object EngineLock [exclusiveLocks= key: 1eaedbbb-1c0e-4028-9b0e-adac7e07d25f value: REGISTER_VDS >, sharedLocks= ] >2012-11-08 09:01:36,955 ERROR [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp-/127.0.0.1:8702-5) [652fe0c6] Cannot get vdsManager for vdsid=cfa543b0-29ac-11e2-b6f4-52540035ed48 >2012-11-08 09:01:36,956 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (ajp-/127.0.0.1:8702-5) [652fe0c6] START, RemoveVdsVDSCommand(HostName = rhs-gp-srv15.lab.eng.blr.redhat.com, HostId = cfa543b0-29ac-11e2-b6f4-52540035ed48), log id: 48223b56 >2012-11-08 09:01:36,956 ERROR [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp-/127.0.0.1:8702-5) [652fe0c6] Cannot get vdsManager for vdsid=cfa543b0-29ac-11e2-b6f4-52540035ed48 >2012-11-08 09:01:36,956 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (ajp-/127.0.0.1:8702-5) [652fe0c6] FINISH, RemoveVdsVDSCommand, log id: 48223b56 >2012-11-08 09:01:36,956 ERROR [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp-/127.0.0.1:8702-5) [652fe0c6] Cannot get vdsManager for vdsid=cfa543b0-29ac-11e2-b6f4-52540035ed48 >2012-11-08 09:01:36,958 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-5) [652fe0c6] START, AddVdsVDSCommand(HostName = rhs-gp-srv15.lab.eng.blr.redhat.com, HostId = cfa543b0-29ac-11e2-b6f4-52540035ed48), log id: 1494e42 >2012-11-08 09:01:36,958 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-5) [652fe0c6] AddVds - entered , starting logic to add VDS cfa543b0-29ac-11e2-b6f4-52540035ed48 >2012-11-08 09:01:36,960 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-5) [652fe0c6] AddVds - VDS cfa543b0-29ac-11e2-b6f4-52540035ed48 was added, will try to add it to the resource manager >2012-11-08 09:01:36,960 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ajp-/127.0.0.1:8702-5) [652fe0c6] Eneterd VdsManager:constructor >2012-11-08 09:01:36,961 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ajp-/127.0.0.1:8702-5) [652fe0c6] vdsBroker(10.70.36.19,54,321) >2012-11-08 09:01:36,962 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp-/127.0.0.1:8702-5) [652fe0c6] ResourceManager::AddVds - VDS cfa543b0-29ac-11e2-b6f4-52540035ed48 was added to the Resource Manager >2012-11-08 09:01:36,962 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-5) [652fe0c6] FINISH, AddVdsVDSCommand, log id: 1494e42 >2012-11-08 09:01:36,978 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-4-thread-10) [10bde595] Running command: InstallVdsCommand internal: true. Entities affected : ID: cfa543b0-29ac-11e2-b6f4-52540035ed48 Type: VDS >2012-11-08 09:01:36,979 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-4-thread-10) [10bde595] Before Installation pool-4-thread-10 >2012-11-08 09:01:36,979 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-10) [10bde595] Installation of 10.70.36.19. Executing installation stage. (Stage: Starting Host installation) >2012-11-08 09:01:36,979 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-10) [10bde595] Installation of 10.70.36.19. Executing installation stage. (Stage: Connecting to Host) >2012-11-08 09:01:37,039 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-10) [10bde595] Installation of 10.70.36.19. Received message: <BSTRAP component='RHEV_INSTALL' status='OK' message='Connected to Host 10.70.36.19 with SSH key fingerprint: 74:1f:5c:d9:6d:8e:d7:72:33:74:37:01:2c:d8:1a:44'/>. FYI. (Stage: Connecting to Host) >2012-11-08 09:01:37,068 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-10) [10bde595] Installation of 10.70.36.19. Successfully connected to server ssh. (Stage: Connecting to Host) >2012-11-08 09:01:37,068 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-10) [10bde595] Installation of 10.70.36.19. Executing installation stage. (Stage: Get the unique vds id) >2012-11-08 09:01:37,068 INFO [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-4-thread-10) SSH execute 10.70.36.19 'IDFILE=/etc/vdsm/vdsm.id; if [ -r "${IDFILE}" ]; then cat "${IDFILE}"; else UUID="$(dmidecode -s system-uuid 2> /dev/null | sed -e 's/.*Not.*//' )"; if [ -z "${UUID}" ]; then UUID="$(uuidgen 2> /dev/null)" && mkdir -p "$(dirname "${IDFILE}")" && echo "${UUID}" > "${IDFILE}" && chmod 0644 "${IDFILE}"; fi; [ -n "${UUID}" ] && echo "${UUID}"; fi' >2012-11-08 09:01:37,177 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-87) Installation of 10.70.36.19. Received message: 00000000-0000-0000-0000-0025902F4FF8. FYI. (Stage: Get the unique vds id) >2012-11-08 09:01:37,186 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-87) Installation of 10.70.36.19. Assigning unique id 00000000-0000-0000-0000-0025902F4FF8 to Host. (Stage: Get the unique vds id) >2012-11-08 09:01:37,189 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-10) [10bde595] Installation of 10.70.36.19. Executing installation stage. (Stage: Upload Installation script to Host) >2012-11-08 09:01:37,190 INFO [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-4-thread-10) SSH execute 10.70.36.19 'cat > '/tmp/ovirt-id_rsa_0d01c420-8f32-498c-9395-1e4026815cdc'' >2012-11-08 09:01:37,275 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-10) [10bde595] Installation of 10.70.36.19. Executing installation stage. (Stage: Running first installation script on Host) >2012-11-08 09:01:37,275 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-10) [10bde595] Installation of 10.70.36.19. Sending SSH Command umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar -C "${MYTMP}" --no-same-permissions -o -x && "${MYTMP}"/setup -c 'ssl=true;management_port=54321' -O 'lab.eng.blr.redhat.com' -t 2012-11-08T14:01:37 -S /tmp/ovirt-id_rsa_0d01c420-8f32-498c-9395-1e4026815cdc -p 80 -b -B rhevm http://guido.lab.eng.blr.redhat.com:80/Components/vds/ http://guido.lab.eng.blr.redhat.com:80/Components/vds/ 10.70.36.19 0d01c420-8f32-498c-9395-1e4026815cdc False < /var/cache/ovirt-engine/vdsm-bootstrap-2.tar. (Stage: Running first installation script on Host) >2012-11-08 09:01:37,275 INFO [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-4-thread-10) SSH execute 10.70.36.19 'umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar -C "${MYTMP}" --no-same-permissions -o -x && "${MYTMP}"/setup -c 'ssl=true;management_port=54321' -O 'lab.eng.blr.redhat.com' -t 2012-11-08T14:01:37 -S /tmp/ovirt-id_rsa_0d01c420-8f32-498c-9395-1e4026815cdc -p 80 -b -B rhevm http://guido.lab.eng.blr.redhat.com:80/Components/vds/ http://guido.lab.eng.blr.redhat.com:80/Components/vds/ 10.70.36.19 0d01c420-8f32-498c-9395-1e4026815cdc False' >2012-11-08 09:01:37,385 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-87) Installation of 10.70.36.19. Received message: <BSTRAP component='INSTALLER' status='OK' message='Test platform succeeded'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:01:38,963 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-22) Initializing Host: rhs-gp-srv15.lab.eng.blr.redhat.com >2012-11-08 09:01:39,487 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-87) Installation of 10.70.36.19. Received message: <BSTRAP component='SET_SYSTEM_TIME' status='OK' message='setSystemTime ended successfully'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:01:41,356 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-87) Installation of 10.70.36.19. Received message: <BSTRAP component='RHN_REGISTRATION' status='OK' message='Host properly registered with RHN/Satellite.'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:01:41,525 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-87) Installation of 10.70.36.19. Received message: <BSTRAP component='VDSM_MAJOR_VER' status='OK' message='Available VDSM matches requirements'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:01:41,530 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-87) Installation of 10.70.36.19. Received message: <BSTRAP component='VT_SVM' status='OK' processor='Intel' message='Server supports virtualization'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:01:41,535 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-87) Installation of 10.70.36.19. Received message: <BSTRAP component='OS' status='OK' type='RHEL6' message='Supported platform version'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:01:41,539 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-87) Installation of 10.70.36.19. Received message: <BSTRAP component='KERNEL' status='OK' version='279' message='Supported kernel version: 279'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:01:41,618 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-87) Installation of 10.70.36.19. Received message: <BSTRAP component='REQ PACKAGES' status='OK' result='SDL.x86_64' message='SDL-1.2.14-3.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:01:41,632 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-87) Installation of 10.70.36.19. Received message: <BSTRAP component='REQ PACKAGES' status='OK' result='bridge-utils.x86_64' message='bridge-utils-1.2-9.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:01:41,647 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-87) Installation of 10.70.36.19. Received message: <BSTRAP component='REQ PACKAGES' status='OK' result='mesa-libGLU.x86_64' message='mesa-libGLU-7.11-5.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:01:41,662 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-87) Installation of 10.70.36.19. Received message: <BSTRAP component='REQ PACKAGES' status='OK' result='openssl.x86_64' message='openssl-1.0.0-25.el6_3.1.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:01:41,677 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-87) Installation of 10.70.36.19. Received message: <BSTRAP component='REQ PACKAGES' status='OK' result='m2crypto.x86_64' message='m2crypto-0.20.2-9.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:01:41,691 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-87) Installation of 10.70.36.19. Received message: <BSTRAP component='REQ PACKAGES' status='OK' result='rsync.x86_64' message='rsync-3.0.6-9.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:01:41,704 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-87) Installation of 10.70.36.19. Received message: <BSTRAP component='VDS PACKAGES' status='WARN' result='qemu-kvm' message='package qemu-kvm is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:01:41,717 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-87) Installation of 10.70.36.19. Received message: <BSTRAP component='VDS PACKAGES' status='WARN' result='qemu-kvm-tools' message='package qemu-kvm-tools is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:01:41,731 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-87) Installation of 10.70.36.19. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='(~vdsm~, ~vdsm-cli~)' message='vdsm-4.9.6-41.0.el6_3.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:01:41,745 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-87) Installation of 10.70.36.19. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='libjpeg' message='libjpeg-6b-46.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:01:41,759 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-87) Installation of 10.70.36.19. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='spice-server' message='spice-server-0.10.1-10.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:01:41,773 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-87) Installation of 10.70.36.19. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='pixman' message='pixman-0.18.4-1.el6_0.1.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:01:41,787 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-87) Installation of 10.70.36.19. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='seabios' message='seabios-0.6.1.2-19.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:01:41,800 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-87) Installation of 10.70.36.19. Received message: <BSTRAP component='VDS PACKAGES' status='WARN' result='qemu-img' message='package qemu-img is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:01:41,814 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-87) Installation of 10.70.36.19. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='fence-agents' message='fence-agents-3.1.5-17.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:01:41,829 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-87) Installation of 10.70.36.19. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='libselinux-python' message='libselinux-python-2.0.94-5.3.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:01:41,842 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-87) Installation of 10.70.36.19. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='sanlock' message='sanlock-2.3-4.el6_3.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:01:41,856 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-87) Installation of 10.70.36.19. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='sanlock-python' message='sanlock-python-2.3-4.el6_3.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:01:59,912 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-73) VM b5 3d766d10-58ab-4306-b5c4-7d109c7b6c46 moved from PoweringUp --> Up >2012-11-08 09:01:59,937 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsDynamicDataVDSCommand] (QuartzScheduler_Worker-73) START, UpdateVdsDynamicDataVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vdsDynamic=org.ovirt.engine.core.common.businessentities.VdsDynamic@4e2c3134), log id: 44475682 >2012-11-08 09:01:59,938 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsDynamicDataVDSCommand] (QuartzScheduler_Worker-73) FINISH, UpdateVdsDynamicDataVDSCommand, log id: 44475682 >2012-11-08 09:02:34,219 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-87) Installation of 10.70.36.19. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='(~vdsm~, ~vdsm-cli~)' message='vdsm-4.9.6-41.0.el6_3.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:02:36,927 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-87) Installation of 10.70.36.19. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='libjpeg' message='libjpeg-6b-46.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:02:39,380 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-87) Installation of 10.70.36.19. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='spice-server' message='spice-server-0.10.1-10.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:02:41,851 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-87) Installation of 10.70.36.19. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='pixman' message='pixman-0.18.4-1.el6_0.1.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:02:44,258 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-87) Installation of 10.70.36.19. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='seabios' message='seabios-0.6.1.2-19.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:02:46,706 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-87) Installation of 10.70.36.19. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='fence-agents' message='fence-agents-3.1.5-17.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:02:49,160 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-87) Installation of 10.70.36.19. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='libselinux-python' message='libselinux-python-2.0.94-5.3.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:02:51,562 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-87) Installation of 10.70.36.19. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='sanlock' message='sanlock-2.3-4.el6_3.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:02:54,004 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-87) Installation of 10.70.36.19. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='sanlock-python' message='sanlock-python-2.3-4.el6_3.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:02:56,451 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-87) Installation of 10.70.36.19. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='qemu-img' message='package qemu-img is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:03:08,409 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-87) Installation of 10.70.36.19. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='qemu-kvm-tools' message='package qemu-kvm-tools is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:03:10,819 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-87) Installation of 10.70.36.19. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='qemu-kvm' message='package qemu-kvm is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:03:10,824 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-87) Installation of 10.70.36.19. Received message: <BSTRAP component='CreateConf' status='OK' message='Basic configuration set'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:03:10,829 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-87) Installation of 10.70.36.19. Received message: <BSTRAP component='CheckLocalHostname' status='OK' message='Local hostname is correct.'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:03:11,614 INFO [org.ovirt.engine.core.bll.SetVmTicketCommand] (ajp-/127.0.0.1:8702-7) [54b631c1] Running command: SetVmTicketCommand internal: false. Entities affected : ID: 3d766d10-58ab-4306-b5c4-7d109c7b6c46 Type: VM >2012-11-08 09:03:11,635 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp-/127.0.0.1:8702-7) [54b631c1] START, SetVmTicketVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=3d766d10-58ab-4306-b5c4-7d109c7b6c46, ticket=ul0hjgy8qE+F, validTime=120,m userName=admin@internal, userId=fdfc627c-d875-11e0-90f0-83df133b58cc), log id: 228a355d >2012-11-08 09:03:11,665 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp-/127.0.0.1:8702-7) [54b631c1] FINISH, SetVmTicketVDSCommand, log id: 228a355d >2012-11-08 09:03:19,684 INFO [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (ajp-/127.0.0.1:8702-10) SSH execute 10.70.36.16 'IDFILE=/etc/vdsm/vdsm.id; if [ -r "${IDFILE}" ]; then cat "${IDFILE}"; else UUID="$(dmidecode -s system-uuid 2> /dev/null | sed -e 's/.*Not.*//' )"; if [ -z "${UUID}" ]; then UUID="$(uuidgen 2> /dev/null)" && mkdir -p "$(dirname "${IDFILE}")" && echo "${UUID}" > "${IDFILE}" && chmod 0644 "${IDFILE}"; fi; [ -n "${UUID}" ] && echo "${UUID}"; fi' >2012-11-08 09:03:19,836 INFO [org.ovirt.engine.core.bll.AddVdsCommand] (ajp-/127.0.0.1:8702-10) [11abc964] Running command: AddVdsCommand internal: false. Entities affected : ID: c04832b4-2976-11e2-b6fb-52540035ed48 Type: VdsGroups >2012-11-08 09:03:19,851 INFO [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp-/127.0.0.1:8702-10) [73d267ed] Running command: AddVdsSpmIdCommand internal: true. Entities affected : ID: 0cfb1244-29ad-11e2-8844-52540035ed48 Type: VDS >2012-11-08 09:03:19,855 INFO [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp-/127.0.0.1:8702-10) [73d267ed] Lock freed to object EngineLock [exclusiveLocks= key: 1eaedbbb-1c0e-4028-9b0e-adac7e07d25f value: REGISTER_VDS >, sharedLocks= ] >2012-11-08 09:03:19,855 ERROR [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp-/127.0.0.1:8702-10) [73d267ed] Cannot get vdsManager for vdsid=0cfb1244-29ad-11e2-8844-52540035ed48 >2012-11-08 09:03:19,857 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (ajp-/127.0.0.1:8702-10) [73d267ed] START, RemoveVdsVDSCommand(HostName = rhs-gp-srv12.lab.eng.blr.redhat.com, HostId = 0cfb1244-29ad-11e2-8844-52540035ed48), log id: 4a16a9b8 >2012-11-08 09:03:19,857 ERROR [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp-/127.0.0.1:8702-10) [73d267ed] Cannot get vdsManager for vdsid=0cfb1244-29ad-11e2-8844-52540035ed48 >2012-11-08 09:03:19,857 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (ajp-/127.0.0.1:8702-10) [73d267ed] FINISH, RemoveVdsVDSCommand, log id: 4a16a9b8 >2012-11-08 09:03:19,857 ERROR [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp-/127.0.0.1:8702-10) [73d267ed] Cannot get vdsManager for vdsid=0cfb1244-29ad-11e2-8844-52540035ed48 >2012-11-08 09:03:19,858 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-10) [73d267ed] START, AddVdsVDSCommand(HostName = rhs-gp-srv12.lab.eng.blr.redhat.com, HostId = 0cfb1244-29ad-11e2-8844-52540035ed48), log id: 8bf882e >2012-11-08 09:03:19,858 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-10) [73d267ed] AddVds - entered , starting logic to add VDS 0cfb1244-29ad-11e2-8844-52540035ed48 >2012-11-08 09:03:19,860 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-10) [73d267ed] AddVds - VDS 0cfb1244-29ad-11e2-8844-52540035ed48 was added, will try to add it to the resource manager >2012-11-08 09:03:19,861 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ajp-/127.0.0.1:8702-10) [73d267ed] Eneterd VdsManager:constructor >2012-11-08 09:03:19,861 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ajp-/127.0.0.1:8702-10) [73d267ed] vdsBroker(10.70.36.16,54,321) >2012-11-08 09:03:19,861 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp-/127.0.0.1:8702-10) [73d267ed] ResourceManager::AddVds - VDS 0cfb1244-29ad-11e2-8844-52540035ed48 was added to the Resource Manager >2012-11-08 09:03:19,862 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-10) [73d267ed] FINISH, AddVdsVDSCommand, log id: 8bf882e >2012-11-08 09:03:19,882 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-4-thread-49) [cb0bc1f] Running command: InstallVdsCommand internal: true. Entities affected : ID: 0cfb1244-29ad-11e2-8844-52540035ed48 Type: VDS >2012-11-08 09:03:19,883 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-4-thread-49) [cb0bc1f] Before Installation pool-4-thread-49 >2012-11-08 09:03:19,883 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-49) [cb0bc1f] Installation of 10.70.36.16. Executing installation stage. (Stage: Starting Host installation) >2012-11-08 09:03:19,883 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-49) [cb0bc1f] Installation of 10.70.36.16. Executing installation stage. (Stage: Connecting to Host) >2012-11-08 09:03:19,946 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-49) [cb0bc1f] Installation of 10.70.36.16. Received message: <BSTRAP component='RHEV_INSTALL' status='OK' message='Connected to Host 10.70.36.16 with SSH key fingerprint: d8:e6:4c:57:0c:54:e1:5a:31:d9:fa:80:44:2a:ac:7f'/>. FYI. (Stage: Connecting to Host) >2012-11-08 09:03:19,976 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-49) [cb0bc1f] Installation of 10.70.36.16. Successfully connected to server ssh. (Stage: Connecting to Host) >2012-11-08 09:03:19,976 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-49) [cb0bc1f] Installation of 10.70.36.16. Executing installation stage. (Stage: Get the unique vds id) >2012-11-08 09:03:19,976 INFO [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-4-thread-49) SSH execute 10.70.36.16 'IDFILE=/etc/vdsm/vdsm.id; if [ -r "${IDFILE}" ]; then cat "${IDFILE}"; else UUID="$(dmidecode -s system-uuid 2> /dev/null | sed -e 's/.*Not.*//' )"; if [ -z "${UUID}" ]; then UUID="$(uuidgen 2> /dev/null)" && mkdir -p "$(dirname "${IDFILE}")" && echo "${UUID}" > "${IDFILE}" && chmod 0644 "${IDFILE}"; fi; [ -n "${UUID}" ] && echo "${UUID}"; fi' >2012-11-08 09:03:20,095 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-92) Installation of 10.70.36.16. Received message: 00000000-0000-0000-0000-0025902F723C. FYI. (Stage: Get the unique vds id) >2012-11-08 09:03:20,099 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-92) Installation of 10.70.36.16. Assigning unique id 00000000-0000-0000-0000-0025902F723C to Host. (Stage: Get the unique vds id) >2012-11-08 09:03:20,102 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-49) [cb0bc1f] Installation of 10.70.36.16. Executing installation stage. (Stage: Upload Installation script to Host) >2012-11-08 09:03:20,102 INFO [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-4-thread-49) SSH execute 10.70.36.16 'cat > '/tmp/ovirt-id_rsa_dbeda2a2-4b13-4740-9888-7cbcc2ebd1d4'' >2012-11-08 09:03:20,186 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-49) [cb0bc1f] Installation of 10.70.36.16. Executing installation stage. (Stage: Running first installation script on Host) >2012-11-08 09:03:20,186 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-49) [cb0bc1f] Installation of 10.70.36.16. Sending SSH Command umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar -C "${MYTMP}" --no-same-permissions -o -x && "${MYTMP}"/setup -c 'ssl=true;management_port=54321' -O 'lab.eng.blr.redhat.com' -t 2012-11-08T14:03:20 -S /tmp/ovirt-id_rsa_dbeda2a2-4b13-4740-9888-7cbcc2ebd1d4 -p 80 -b -B rhevm http://guido.lab.eng.blr.redhat.com:80/Components/vds/ http://guido.lab.eng.blr.redhat.com:80/Components/vds/ 10.70.36.16 dbeda2a2-4b13-4740-9888-7cbcc2ebd1d4 False < /var/cache/ovirt-engine/vdsm-bootstrap-2.tar. (Stage: Running first installation script on Host) >2012-11-08 09:03:20,187 INFO [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-4-thread-49) SSH execute 10.70.36.16 'umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar -C "${MYTMP}" --no-same-permissions -o -x && "${MYTMP}"/setup -c 'ssl=true;management_port=54321' -O 'lab.eng.blr.redhat.com' -t 2012-11-08T14:03:20 -S /tmp/ovirt-id_rsa_dbeda2a2-4b13-4740-9888-7cbcc2ebd1d4 -p 80 -b -B rhevm http://guido.lab.eng.blr.redhat.com:80/Components/vds/ http://guido.lab.eng.blr.redhat.com:80/Components/vds/ 10.70.36.16 dbeda2a2-4b13-4740-9888-7cbcc2ebd1d4 False' >2012-11-08 09:03:20,298 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-92) Installation of 10.70.36.16. Received message: <BSTRAP component='INSTALLER' status='OK' message='Test platform succeeded'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:03:21,404 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-92) Installation of 10.70.36.16. Received message: <BSTRAP component='SET_SYSTEM_TIME' status='OK' message='setSystemTime ended successfully'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:03:21,863 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-73) Initializing Host: rhs-gp-srv12.lab.eng.blr.redhat.com >2012-11-08 09:03:23,215 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-92) Installation of 10.70.36.16. Received message: <BSTRAP component='RHN_REGISTRATION' status='OK' message='Host properly registered with RHN/Satellite.'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:03:23,385 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-92) Installation of 10.70.36.16. Received message: <BSTRAP component='VDSM_MAJOR_VER' status='OK' message='Available VDSM matches requirements'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:03:23,389 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-92) Installation of 10.70.36.16. Received message: <BSTRAP component='VT_SVM' status='OK' processor='Intel' message='Server supports virtualization'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:03:23,392 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-92) Installation of 10.70.36.16. Received message: <BSTRAP component='OS' status='OK' type='RHEL6' message='Supported platform version'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:03:23,395 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-92) Installation of 10.70.36.16. Received message: <BSTRAP component='KERNEL' status='OK' version='279' message='Supported kernel version: 279'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:03:23,478 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-92) Installation of 10.70.36.16. Received message: <BSTRAP component='REQ PACKAGES' status='OK' result='SDL.x86_64' message='SDL-1.2.14-3.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:03:23,493 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-92) Installation of 10.70.36.16. Received message: <BSTRAP component='REQ PACKAGES' status='OK' result='bridge-utils.x86_64' message='bridge-utils-1.2-9.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:03:23,507 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-92) Installation of 10.70.36.16. Received message: <BSTRAP component='REQ PACKAGES' status='OK' result='mesa-libGLU.x86_64' message='mesa-libGLU-7.11-5.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:03:23,522 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-92) Installation of 10.70.36.16. Received message: <BSTRAP component='REQ PACKAGES' status='OK' result='openssl.x86_64' message='openssl-1.0.0-25.el6_3.1.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:03:23,537 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-92) Installation of 10.70.36.16. Received message: <BSTRAP component='REQ PACKAGES' status='OK' result='m2crypto.x86_64' message='m2crypto-0.20.2-9.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:03:23,551 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-92) Installation of 10.70.36.16. Received message: <BSTRAP component='REQ PACKAGES' status='OK' result='rsync.x86_64' message='rsync-3.0.6-9.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:03:23,564 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-92) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='WARN' result='qemu-kvm' message='package qemu-kvm is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:03:23,577 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-92) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='WARN' result='qemu-kvm-tools' message='package qemu-kvm-tools is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:03:23,591 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-92) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='(~vdsm~, ~vdsm-cli~)' message='vdsm-4.9.6-41.0.el6_3.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:03:23,605 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-92) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='libjpeg' message='libjpeg-6b-46.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:03:23,619 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-92) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='spice-server' message='spice-server-0.10.1-10.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:03:23,633 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-92) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='pixman' message='pixman-0.18.4-1.el6_0.1.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:03:23,647 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-92) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='seabios' message='seabios-0.6.1.2-19.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:03:23,660 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-92) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='WARN' result='qemu-img' message='package qemu-img is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:03:23,674 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-92) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='fence-agents' message='fence-agents-3.1.5-17.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:03:23,689 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-92) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='libselinux-python' message='libselinux-python-2.0.94-5.3.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:03:23,702 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-92) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='sanlock' message='sanlock-2.3-4.el6_3.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:03:23,716 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-92) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='sanlock-python' message='sanlock-python-2.3-4.el6_3.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:03:25,387 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-87) Installation of 10.70.36.19. Received message: <BSTRAP component='SetNetworking' status='OK' message='Created management bridge.'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:03:25,498 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-87) Installation of 10.70.36.19. Received message: <BSTRAP component='SetSSHAccess' status='OK' message='SUCCESS'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:03:25,778 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-87) Installation of 10.70.36.19. Received message: <BSTRAP component='VerifyServices' status='OK' message='Needed services set'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:03:34,474 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-87) Installation of 10.70.36.19. Received message: <BSTRAP component='Encryption setup' status='OK' message='Ended successfully'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:03:34,480 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-87) Installation of 10.70.36.19. Received message: <BSTRAP component='RHEV_INSTALL' status='OK'/>. Stage completed. (Stage: Running first installation script on Host) >2012-11-08 09:03:34,515 ERROR [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-10) [10bde595] Installation of 10.70.36.19. Error: Repository JBOSS is listed more than once in the configuration >Repository qa-latest-5 is listed more than once in the configuration >Repository rhel-source is listed more than once in the configuration >Repository rhel-source-beta is listed more than once in the configuration >Repository rhs-client is listed more than once in the configuration >Repository JBOSS is listed more than once in the configuration >Repository qa-latest-5 is listed more than once in the configuration >Repository rhel-source is listed more than once in the configuration >Repository rhel-source-beta is listed more than once in the configuration >Repository rhs-client is listed more than once in the configuration >. (Stage: Downloading certificate request from Host) >2012-11-08 09:03:34,515 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-10) [10bde595] VDS message: Repository JBOSS is listed more than once in the configuration >2012-11-08 09:03:34,515 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-10) [10bde595] VDS message: Repository qa-latest-5 is listed more than once in the configuration >2012-11-08 09:03:34,515 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-10) [10bde595] VDS message: Repository rhel-source is listed more than once in the configuration >2012-11-08 09:03:34,515 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-10) [10bde595] VDS message: Repository rhel-source-beta is listed more than once in the configuration >2012-11-08 09:03:34,515 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-10) [10bde595] VDS message: Repository rhs-client is listed more than once in the configuration >2012-11-08 09:03:34,515 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-10) [10bde595] VDS message: Repository JBOSS is listed more than once in the configuration >2012-11-08 09:03:34,515 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-10) [10bde595] VDS message: Repository qa-latest-5 is listed more than once in the configuration >2012-11-08 09:03:34,515 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-10) [10bde595] VDS message: Repository rhel-source is listed more than once in the configuration >2012-11-08 09:03:34,515 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-10) [10bde595] VDS message: Repository rhel-source-beta is listed more than once in the configuration >2012-11-08 09:03:34,515 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-10) [10bde595] VDS message: Repository rhs-client is listed more than once in the configuration >2012-11-08 09:03:34,515 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-10) [10bde595] Script ended, result is {1} >2012-11-08 09:03:34,515 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-10) [10bde595] Installation of 10.70.36.19. Executing installation stage. (Stage: Downloading certificate request from Host) >2012-11-08 09:03:34,515 INFO [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-4-thread-10) SSH receive 10.70.36.19:'/tmp/cert_0d01c420-8f32-498c-9395-1e4026815cdc.req' '/etc/pki/ovirt-engine/requests/cert_0d01c420-8f32-498c-9395-1e4026815cdc.req') >2012-11-08 09:03:34,577 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-10) [10bde595] Installation of 10.70.36.19. successfully done sftp operation ( Stage: Downloading certificate request from Host) >2012-11-08 09:03:34,577 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-10) [10bde595] DownloadCertificateRequest ended:true >2012-11-08 09:03:34,577 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-10) [10bde595] Installation of 10.70.36.19. Executing installation stage. (Stage: Sign certificate request and generate certificate) >2012-11-08 09:03:35,586 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-10) [10bde595] SignCertificateRequest ended:true >2012-11-08 09:03:35,586 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-10) [10bde595] Installation of 10.70.36.19. Executing installation stage. (Stage: Upload signed sertificate to Host) >2012-11-08 09:03:35,610 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-10) [10bde595] Installation of 10.70.36.19. successfully done sftp operation ( Stage: Upload signed sertificate to Host) >2012-11-08 09:03:35,610 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-10) [10bde595] UploadSignedCertificate ended:true >2012-11-08 09:03:35,610 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-10) [10bde595] Installation of 10.70.36.19. Executing installation stage. (Stage: Upload Cerficate Autority to Host) >2012-11-08 09:03:35,665 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-10) [10bde595] Installation of 10.70.36.19. successfully done sftp operation ( Stage: Upload Cerficate Autority to Host) >2012-11-08 09:03:35,665 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-10) [10bde595] Installation of 10.70.36.19. Executing installation stage. (Stage: Running second installation script on Host) >2012-11-08 09:03:35,665 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-10) [10bde595] Installation of 10.70.36.19. Sending SSH Command umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar -C "${MYTMP}" --no-same-permissions -o -x && "${MYTMP}"/setup -c 'ssl=true;management_port=54321' -O 'lab.eng.blr.redhat.com' -t 2012-11-08T14:01:37 -S /tmp/ovirt-id_rsa_0d01c420-8f32-498c-9395-1e4026815cdc -p 80 -b -B rhevm http://guido.lab.eng.blr.redhat.com:80/Components/vds/ http://guido.lab.eng.blr.redhat.com:80/Components/vds/ 10.70.36.19 0d01c420-8f32-498c-9395-1e4026815cdc True < /var/cache/ovirt-engine/vdsm-bootstrap-2.tar. (Stage: Running second installation script on Host) >2012-11-08 09:03:35,666 INFO [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-4-thread-10) SSH execute 10.70.36.19 'umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar -C "${MYTMP}" --no-same-permissions -o -x && "${MYTMP}"/setup -c 'ssl=true;management_port=54321' -O 'lab.eng.blr.redhat.com' -t 2012-11-08T14:01:37 -S /tmp/ovirt-id_rsa_0d01c420-8f32-498c-9395-1e4026815cdc -p 80 -b -B rhevm http://guido.lab.eng.blr.redhat.com:80/Components/vds/ http://guido.lab.eng.blr.redhat.com:80/Components/vds/ 10.70.36.19 0d01c420-8f32-498c-9395-1e4026815cdc True' >2012-11-08 09:03:35,756 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-87) Installation of 10.70.36.19. Received message: <BSTRAP component='INSTALLER' status='OK' message='Test platform succeeded'/>. FYI. (Stage: Running second installation script on Host) >2012-11-08 09:03:36,067 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-87) Installation of 10.70.36.19. Received message: <BSTRAP component='instCert' status='OK'/>. FYI. (Stage: Running second installation script on Host) >2012-11-08 09:03:36,070 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-87) Installation of 10.70.36.19. Received message: <BSTRAP component='CoreDump' status='OK'/>. FYI. (Stage: Running second installation script on Host) >2012-11-08 09:03:36,087 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-87) Installation of 10.70.36.19. Received message: <BSTRAP component='cleanAll' status='OK'/>. FYI. (Stage: Running second installation script on Host) >2012-11-08 09:03:36,092 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-87) Installation of 10.70.36.19. Received message: <BSTRAP component='VDS Configuration' status='OK'/>. FYI. (Stage: Running second installation script on Host) >2012-11-08 09:03:36,096 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-87) Installation of 10.70.36.19. Received message: <BSTRAP component='RHEV_INSTALL' status='OK'/>. Stage completed. (Stage: Running second installation script on Host) >2012-11-08 09:03:36,100 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-87) Installation of 10.70.36.19. Received message: <BSTRAP component='Reboot' status='OK' message='Rebooting machine' />. FYI. (Stage: Host installation complete) >2012-11-08 09:03:36,110 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-10) [10bde595] Script ended, result is {1} >2012-11-08 09:03:36,111 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-4-thread-10) [10bde595] After Installation pool-4-thread-10 >2012-11-08 09:03:36,114 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-10) [10bde595] START, SetVdsStatusVDSCommand(HostName = rhs-gp-srv15.lab.eng.blr.redhat.com, HostId = cfa543b0-29ac-11e2-b6f4-52540035ed48, status=Reboot, nonOperationalReason=NONE), log id: 4ccc026b >2012-11-08 09:03:36,123 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-10) [10bde595] FINISH, SetVdsStatusVDSCommand, log id: 4ccc026b >2012-11-08 09:03:36,123 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-4-thread-47) [10bde595] Waiting 300 seconds, for server to finish reboot process. >2012-11-08 09:04:12,232 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-92) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='(~vdsm~, ~vdsm-cli~)' message='vdsm-4.9.6-41.0.el6_3.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:04:14,947 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-92) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='libjpeg' message='libjpeg-6b-46.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:04:17,722 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-92) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='spice-server' message='spice-server-0.10.1-10.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:04:20,124 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-92) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='pixman' message='pixman-0.18.4-1.el6_0.1.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:04:22,524 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-92) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='seabios' message='seabios-0.6.1.2-19.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:04:24,963 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-92) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='fence-agents' message='fence-agents-3.1.5-17.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:04:27,444 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-92) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='libselinux-python' message='libselinux-python-2.0.94-5.3.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:04:29,852 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-92) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='sanlock' message='sanlock-2.3-4.el6_3.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:04:32,295 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-92) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='sanlock-python' message='sanlock-python-2.3-4.el6_3.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:04:34,748 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-92) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='qemu-img' message='package qemu-img is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:04:37,207 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-92) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='qemu-kvm-tools' message='package qemu-kvm-tools is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:04:39,663 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-92) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='qemu-kvm' message='package qemu-kvm is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:04:39,667 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-92) Installation of 10.70.36.16. Received message: <BSTRAP component='CreateConf' status='OK' message='Basic configuration found, skipping this step'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:04:39,669 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-92) Installation of 10.70.36.16. Received message: <BSTRAP component='CheckLocalHostname' status='OK' message='Local hostname is correct.'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:04:39,727 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-92) Installation of 10.70.36.16. Received message: <BSTRAP component='SetNetworking' status='OK' message='Bridge management already exists. Skipping bridge creation.'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:04:39,848 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-92) Installation of 10.70.36.16. Received message: <BSTRAP component='SetSSHAccess' status='OK' message='SUCCESS'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:04:40,039 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-92) Installation of 10.70.36.16. Received message: <BSTRAP component='VerifyServices' status='OK' message='Needed services set'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:04:42,755 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-92) Installation of 10.70.36.16. Received message: <BSTRAP component='Encryption setup' status='OK' message='Ended successfully'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:04:42,760 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-92) Installation of 10.70.36.16. Received message: <BSTRAP component='RHEV_INSTALL' status='OK'/>. Stage completed. (Stage: Running first installation script on Host) >2012-11-08 09:04:42,790 ERROR [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-49) [cb0bc1f] Installation of 10.70.36.16. Error: Repository JBOSS is listed more than once in the configuration >Repository qa-latest-5 is listed more than once in the configuration >Repository rhel-source is listed more than once in the configuration >Repository rhel-source-beta is listed more than once in the configuration >Repository rhs-client is listed more than once in the configuration >Repository JBOSS is listed more than once in the configuration >Repository qa-latest-5 is listed more than once in the configuration >Repository rhel-source is listed more than once in the configuration >Repository rhel-source-beta is listed more than once in the configuration >Repository rhs-client is listed more than once in the configuration >. (Stage: Downloading certificate request from Host) >2012-11-08 09:04:42,790 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-49) [cb0bc1f] VDS message: Repository JBOSS is listed more than once in the configuration >2012-11-08 09:04:42,790 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-49) [cb0bc1f] VDS message: Repository qa-latest-5 is listed more than once in the configuration >2012-11-08 09:04:42,790 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-49) [cb0bc1f] VDS message: Repository rhel-source is listed more than once in the configuration >2012-11-08 09:04:42,790 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-49) [cb0bc1f] VDS message: Repository rhel-source-beta is listed more than once in the configuration >2012-11-08 09:04:42,790 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-49) [cb0bc1f] VDS message: Repository rhs-client is listed more than once in the configuration >2012-11-08 09:04:42,790 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-49) [cb0bc1f] VDS message: Repository JBOSS is listed more than once in the configuration >2012-11-08 09:04:42,790 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-49) [cb0bc1f] VDS message: Repository qa-latest-5 is listed more than once in the configuration >2012-11-08 09:04:42,790 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-49) [cb0bc1f] VDS message: Repository rhel-source is listed more than once in the configuration >2012-11-08 09:04:42,790 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-49) [cb0bc1f] VDS message: Repository rhel-source-beta is listed more than once in the configuration >2012-11-08 09:04:42,790 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-49) [cb0bc1f] VDS message: Repository rhs-client is listed more than once in the configuration >2012-11-08 09:04:42,790 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-49) [cb0bc1f] Script ended, result is {1} >2012-11-08 09:04:42,790 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-49) [cb0bc1f] Installation of 10.70.36.16. Executing installation stage. (Stage: Downloading certificate request from Host) >2012-11-08 09:04:42,790 INFO [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-4-thread-49) SSH receive 10.70.36.16:'/tmp/cert_dbeda2a2-4b13-4740-9888-7cbcc2ebd1d4.req' '/etc/pki/ovirt-engine/requests/cert_dbeda2a2-4b13-4740-9888-7cbcc2ebd1d4.req') >2012-11-08 09:04:42,862 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-49) [cb0bc1f] Installation of 10.70.36.16. successfully done sftp operation ( Stage: Downloading certificate request from Host) >2012-11-08 09:04:42,862 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-49) [cb0bc1f] DownloadCertificateRequest ended:true >2012-11-08 09:04:42,862 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-49) [cb0bc1f] Installation of 10.70.36.16. Executing installation stage. (Stage: Sign certificate request and generate certificate) >2012-11-08 09:04:43,864 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-49) [cb0bc1f] SignCertificateRequest ended:true >2012-11-08 09:04:43,864 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-49) [cb0bc1f] Installation of 10.70.36.16. Executing installation stage. (Stage: Upload signed sertificate to Host) >2012-11-08 09:04:43,885 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-49) [cb0bc1f] Installation of 10.70.36.16. successfully done sftp operation ( Stage: Upload signed sertificate to Host) >2012-11-08 09:04:43,885 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-49) [cb0bc1f] UploadSignedCertificate ended:true >2012-11-08 09:04:43,885 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-49) [cb0bc1f] Installation of 10.70.36.16. Executing installation stage. (Stage: Upload Cerficate Autority to Host) >2012-11-08 09:04:43,940 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-49) [cb0bc1f] Installation of 10.70.36.16. successfully done sftp operation ( Stage: Upload Cerficate Autority to Host) >2012-11-08 09:04:43,940 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-49) [cb0bc1f] Installation of 10.70.36.16. Executing installation stage. (Stage: Running second installation script on Host) >2012-11-08 09:04:43,940 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-49) [cb0bc1f] Installation of 10.70.36.16. Sending SSH Command umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar -C "${MYTMP}" --no-same-permissions -o -x && "${MYTMP}"/setup -c 'ssl=true;management_port=54321' -O 'lab.eng.blr.redhat.com' -t 2012-11-08T14:03:20 -S /tmp/ovirt-id_rsa_dbeda2a2-4b13-4740-9888-7cbcc2ebd1d4 -p 80 -b -B rhevm http://guido.lab.eng.blr.redhat.com:80/Components/vds/ http://guido.lab.eng.blr.redhat.com:80/Components/vds/ 10.70.36.16 dbeda2a2-4b13-4740-9888-7cbcc2ebd1d4 True < /var/cache/ovirt-engine/vdsm-bootstrap-2.tar. (Stage: Running second installation script on Host) >2012-11-08 09:04:43,941 INFO [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-4-thread-49) SSH execute 10.70.36.16 'umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar -C "${MYTMP}" --no-same-permissions -o -x && "${MYTMP}"/setup -c 'ssl=true;management_port=54321' -O 'lab.eng.blr.redhat.com' -t 2012-11-08T14:03:20 -S /tmp/ovirt-id_rsa_dbeda2a2-4b13-4740-9888-7cbcc2ebd1d4 -p 80 -b -B rhevm http://guido.lab.eng.blr.redhat.com:80/Components/vds/ http://guido.lab.eng.blr.redhat.com:80/Components/vds/ 10.70.36.16 dbeda2a2-4b13-4740-9888-7cbcc2ebd1d4 True' >2012-11-08 09:04:44,030 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-92) Installation of 10.70.36.16. Received message: <BSTRAP component='INSTALLER' status='OK' message='Test platform succeeded'/>. FYI. (Stage: Running second installation script on Host) >2012-11-08 09:04:44,356 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-92) Installation of 10.70.36.16. Received message: <BSTRAP component='instCert' status='OK'/>. FYI. (Stage: Running second installation script on Host) >2012-11-08 09:04:44,361 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-92) Installation of 10.70.36.16. Received message: <BSTRAP component='CoreDump' status='OK'/>. FYI. (Stage: Running second installation script on Host) >2012-11-08 09:04:44,365 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-92) Installation of 10.70.36.16. Received message: <BSTRAP component='cleanAll' status='OK'/>. FYI. (Stage: Running second installation script on Host) >2012-11-08 09:04:44,370 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-92) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS Configuration' status='OK'/>. FYI. (Stage: Running second installation script on Host) >2012-11-08 09:04:44,374 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-92) Installation of 10.70.36.16. Received message: <BSTRAP component='RHEV_INSTALL' status='OK'/>. Stage completed. (Stage: Running second installation script on Host) >2012-11-08 09:04:44,389 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-92) Installation of 10.70.36.16. Received message: <BSTRAP component='Reboot' status='OK' message='Rebooting machine' />. FYI. (Stage: Host installation complete) >2012-11-08 09:04:44,405 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-49) [cb0bc1f] Script ended, result is {1} >2012-11-08 09:04:44,406 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-4-thread-49) [cb0bc1f] After Installation pool-4-thread-49 >2012-11-08 09:04:44,408 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-49) [cb0bc1f] START, SetVdsStatusVDSCommand(HostName = rhs-gp-srv12.lab.eng.blr.redhat.com, HostId = 0cfb1244-29ad-11e2-8844-52540035ed48, status=Reboot, nonOperationalReason=NONE), log id: 6941b67c >2012-11-08 09:04:44,414 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-49) [cb0bc1f] FINISH, SetVdsStatusVDSCommand, log id: 6941b67c >2012-11-08 09:04:44,414 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-4-thread-45) [cb0bc1f] Waiting 300 seconds, for server to finish reboot process. >2012-11-08 09:05:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-46) Autorecovering hosts is disabled, skipping >2012-11-08 09:05:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-46) Autorecovering storage domains is disabled, skipping >2012-11-08 09:08:36,125 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-47) [10bde595] START, SetVdsStatusVDSCommand(HostName = rhs-gp-srv15.lab.eng.blr.redhat.com, HostId = cfa543b0-29ac-11e2-b6f4-52540035ed48, status=NonResponsive, nonOperationalReason=NONE), log id: 1cedbc55 >2012-11-08 09:08:36,130 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-47) [10bde595] FINISH, SetVdsStatusVDSCommand, log id: 1cedbc55 >2012-11-08 09:08:37,363 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-95) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:08:39,372 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-1) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:08:41,380 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-2) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:08:43,400 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-4) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:08:45,416 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-23) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:08:47,429 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-21) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:08:49,440 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-39) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:08:51,449 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-32) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:08:53,460 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-45) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:08:55,472 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-52) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:08:57,485 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-57) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:08:59,499 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-58) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:09:01,512 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-64) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:09:03,524 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-73) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:09:05,534 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-90) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:09:07,544 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-86) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:09:09,554 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-84) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:09:11,565 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-5) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:09:13,591 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-1) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:09:15,603 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-7) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:09:17,616 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-18) [7ab5f38b] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:09:19,629 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-23) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:09:21,641 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-92) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:09:23,652 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-39) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:09:25,668 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-33) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:09:25,669 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Checking if user admin@internal is an admin, result true >2012-11-08 09:09:25,669 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2012-11-08 09:09:25,676 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2012-11-08 09:09:25,676 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2012-11-08 09:09:44,416 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-45) [cb0bc1f] START, SetVdsStatusVDSCommand(HostName = rhs-gp-srv12.lab.eng.blr.redhat.com, HostId = 0cfb1244-29ad-11e2-8844-52540035ed48, status=NonResponsive, nonOperationalReason=NONE), log id: 666c13eb >2012-11-08 09:09:44,420 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-45) [cb0bc1f] FINISH, SetVdsStatusVDSCommand, log id: 666c13eb >2012-11-08 09:10:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-13) Autorecovering hosts is disabled, skipping >2012-11-08 09:10:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-13) Autorecovering storage domains is disabled, skipping >2012-11-08 09:10:51,816 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-16) Checking if user admin@internal is an admin, result true >2012-11-08 09:10:51,816 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-16) Running command: LoginUserCommand internal: false. >2012-11-08 09:10:51,825 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-16) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2012-11-08 09:10:51,826 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-16) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2012-11-08 09:10:52,059 INFO [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (ajp-/127.0.0.1:8702-1) [5beb7f72] Lock Acquired to object EngineLock [exclusiveLocks= key: ms_vm_auto value: VM_NAME >, sharedLocks= ] >2012-11-08 09:10:52,078 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-1) [5beb7f72] START, IsValidVDSCommand( storagePoolId = f4c98a0b-0578-485f-90f4-743df3d05d7e, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 79a83498 >2012-11-08 09:10:52,080 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-1) [5beb7f72] FINISH, IsValidVDSCommand, return: true, log id: 79a83498 >2012-11-08 09:10:52,086 INFO [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (ajp-/127.0.0.1:8702-1) [5beb7f72] Running command: AddVmFromScratchCommand internal: false. Entities affected : ID: f9f22222-2976-11e2-a2cd-52540035ed48 Type: VdsGroups >2012-11-08 09:10:52,089 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-1) [5beb7f72] START, IsValidVDSCommand( storagePoolId = f4c98a0b-0578-485f-90f4-743df3d05d7e, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 3c9f3cc2 >2012-11-08 09:10:52,094 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-1) [5beb7f72] FINISH, IsValidVDSCommand, return: true, log id: 3c9f3cc2 >2012-11-08 09:10:52,108 INFO [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (ajp-/127.0.0.1:8702-1) [5beb7f72] Lock freed to object EngineLock [exclusiveLocks= key: ms_vm_auto value: VM_NAME >, sharedLocks= ] >2012-11-08 09:11:38,699 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Checking if user admin@internal is an admin, result true >2012-11-08 09:11:38,699 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2012-11-08 09:11:38,705 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2012-11-08 09:11:38,706 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2012-11-08 09:12:10,989 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-12) Checking if user admin@internal is an admin, result true >2012-11-08 09:12:10,989 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-12) Running command: LoginUserCommand internal: false. >2012-11-08 09:12:10,995 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-12) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2012-11-08 09:12:10,998 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-12) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2012-11-08 09:12:11,366 WARN [org.ovirt.engine.core.bll.RunVmOnceCommand] (ajp-/127.0.0.1:8702-11) [70f163d9] CanDoAction of action RunVmOnce failed. Reasons:VAR__ACTION__RUN,VAR__TYPE__VM,VM_CANNOT_RUN_FROM_DISK_WITHOUT_DISK >2012-11-08 09:12:11,369 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (ajp-/127.0.0.1:8702-11) Operation Failed: [Cannot run VM without at least one bootable disk. >Alternatives: >-Create a disk for this VM, and rerun the VM. >-Change the boot sequence using the Edit VM command (Boot Option Sub-Tab). >-Use the Run-Once command to select a different boot option and rerun the VM.] >2012-11-08 09:12:13,838 INFO [org.ovirt.engine.core.bll.MaintananceNumberOfVdssCommand] (pool-4-thread-10) [6b861903] Running command: MaintananceNumberOfVdssCommand internal: false. Entities affected : ID: 0cfb1244-29ad-11e2-8844-52540035ed48 Type: VDS >2012-11-08 09:12:13,849 INFO [org.ovirt.engine.core.bll.MaintananceVdsCommand] (pool-4-thread-10) [6b861903] Running command: MaintananceVdsCommand internal: true. Entities affected : ID: 0cfb1244-29ad-11e2-8844-52540035ed48 Type: VDS >2012-11-08 09:12:13,850 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-10) [6b861903] START, SetVdsStatusVDSCommand(HostName = rhs-gp-srv12.lab.eng.blr.redhat.com, HostId = 0cfb1244-29ad-11e2-8844-52540035ed48, status=Maintenance, nonOperationalReason=NONE), log id: 360f22fa >2012-11-08 09:12:16,780 INFO [org.ovirt.engine.core.bll.MaintananceNumberOfVdssCommand] (pool-4-thread-50) [626753a2] Running command: MaintananceNumberOfVdssCommand internal: false. Entities affected : ID: cfa543b0-29ac-11e2-b6f4-52540035ed48 Type: VDS >2012-11-08 09:12:16,832 INFO [org.ovirt.engine.core.bll.MaintananceVdsCommand] (pool-4-thread-50) [626753a2] Running command: MaintananceVdsCommand internal: true. Entities affected : ID: cfa543b0-29ac-11e2-b6f4-52540035ed48 Type: VDS >2012-11-08 09:12:16,834 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-50) [626753a2] START, SetVdsStatusVDSCommand(HostName = rhs-gp-srv15.lab.eng.blr.redhat.com, HostId = cfa543b0-29ac-11e2-b6f4-52540035ed48, status=Maintenance, nonOperationalReason=NONE), log id: ba43eeb >2012-11-08 09:12:27,680 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-51) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.TimeoutException, TimeoutException: >2012-11-08 09:12:27,687 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-50) [626753a2] FINISH, SetVdsStatusVDSCommand, log id: ba43eeb >2012-11-08 09:12:30,244 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-15) Checking if user admin@internal is an admin, result true >2012-11-08 09:12:30,244 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-15) Running command: LoginUserCommand internal: false. >2012-11-08 09:12:30,250 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-15) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2012-11-08 09:12:30,250 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-15) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2012-11-08 09:12:37,223 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-13) Checking if user admin@internal is an admin, result true >2012-11-08 09:12:37,223 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-13) Running command: LoginUserCommand internal: false. >2012-11-08 09:12:37,233 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-13) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2012-11-08 09:12:37,233 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-13) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2012-11-08 09:12:37,566 INFO [org.ovirt.engine.core.bll.RemoveVmCommand] (ajp-/127.0.0.1:8702-14) [4d34f202] Lock Acquired to object EngineLock [exclusiveLocks= key: b2be18de-326c-4713-844b-b47932998110 value: VM >, sharedLocks= ] >2012-11-08 09:12:37,571 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-14) [4d34f202] START, IsValidVDSCommand( storagePoolId = f4c98a0b-0578-485f-90f4-743df3d05d7e, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 27e660fa >2012-11-08 09:12:37,572 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-14) [4d34f202] FINISH, IsValidVDSCommand, return: true, log id: 27e660fa >2012-11-08 09:12:37,577 INFO [org.ovirt.engine.core.bll.RemoveVmCommand] (ajp-/127.0.0.1:8702-14) [4d34f202] Running command: RemoveVmCommand internal: false. Entities affected : ID: b2be18de-326c-4713-844b-b47932998110 Type: VM >2012-11-08 09:12:37,578 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (ajp-/127.0.0.1:8702-14) [4d34f202] START, SetVmStatusVDSCommand( vmId = b2be18de-326c-4713-844b-b47932998110, status = ImageLocked), log id: 567f8fc3 >2012-11-08 09:12:37,581 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (ajp-/127.0.0.1:8702-14) [4d34f202] FINISH, SetVmStatusVDSCommand, log id: 567f8fc3 >2012-11-08 09:12:37,584 INFO [org.ovirt.engine.core.bll.RemoveVmCommand] (ajp-/127.0.0.1:8702-14) [4d34f202] Lock freed to object EngineLock [exclusiveLocks= key: b2be18de-326c-4713-844b-b47932998110 value: VM >, sharedLocks= ] >2012-11-08 09:12:37,585 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.RemoveVMVDSCommand] (ajp-/127.0.0.1:8702-14) [4d34f202] START, RemoveVMVDSCommand( storagePoolId = f4c98a0b-0578-485f-90f4-743df3d05d7e, ignoreFailoverLimit = false, compatabilityVersion = null, storageDomainId = 00000000-0000-0000-0000-000000000000, vmGuid = b2be18de-326c-4713-844b-b47932998110), log id: 6e3df8c5 >2012-11-08 09:12:37,614 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.RemoveVMVDSCommand] (ajp-/127.0.0.1:8702-14) [4d34f202] FINISH, RemoveVMVDSCommand, log id: 6e3df8c5 >2012-11-08 09:12:44,974 WARN [org.ovirt.engine.core.bll.MaintananceNumberOfVdssCommand] (ajp-/127.0.0.1:8702-14) CanDoAction of action MaintananceNumberOfVdss failed. Reasons:VAR__TYPE__HOST,VAR__ACTION__MAINTENANCE,VDS_CANNOT_MAINTENANCE_VDS_IS_IN_MAINTENANCE >2012-11-08 09:12:45,225 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-12) Checking if user admin@internal is an admin, result true >2012-11-08 09:12:45,225 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-12) Running command: LoginUserCommand internal: false. >2012-11-08 09:12:45,229 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-12) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2012-11-08 09:12:45,230 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-12) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2012-11-08 09:12:46,207 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-94) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.TimeoutException, TimeoutException: >2012-11-08 09:12:46,211 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-10) [6b861903] FINISH, SetVdsStatusVDSCommand, log id: 360f22fa >2012-11-08 09:15:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-52) Autorecovering hosts is disabled, skipping >2012-11-08 09:15:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-52) Autorecovering storage domains is disabled, skipping >2012-11-08 09:16:09,392 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (ajp-/127.0.0.1:8702-12) Lock Acquired to object EngineLock [exclusiveLocks= key: 0cfb1244-29ad-11e2-8844-52540035ed48 value: VDS >, sharedLocks= ] >2012-11-08 09:16:09,403 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (pool-4-thread-10) [5fb30edb] Running command: ActivateVdsCommand internal: false. Entities affected : ID: 0cfb1244-29ad-11e2-8844-52540035ed48 Type: VDS >2012-11-08 09:16:09,410 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-10) [5fb30edb] START, SetVdsStatusVDSCommand(HostName = rhs-gp-srv12.lab.eng.blr.redhat.com, HostId = 0cfb1244-29ad-11e2-8844-52540035ed48, status=Unassigned, nonOperationalReason=NONE), log id: 6f9ca535 >2012-11-08 09:16:09,412 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-10) [5fb30edb] FINISH, SetVdsStatusVDSCommand, log id: 6f9ca535 >2012-11-08 09:16:09,414 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (pool-4-thread-10) [5fb30edb] START, ActivateVdsVDSCommand(HostName = rhs-gp-srv12.lab.eng.blr.redhat.com, HostId = 0cfb1244-29ad-11e2-8844-52540035ed48), log id: 506162a6 >2012-11-08 09:16:09,417 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (pool-4-thread-10) [5fb30edb] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:16:09,419 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (pool-4-thread-10) [5fb30edb] ResourceManager::activateVds - failed to get VDS = 0cfb1244-29ad-11e2-8844-52540035ed48 capabilities with error: VDSNetworkException: . >2012-11-08 09:16:09,419 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (pool-4-thread-10) [5fb30edb] ResourceManager::activateVds - failed to activate VDS = 0cfb1244-29ad-11e2-8844-52540035ed48 >2012-11-08 09:16:09,421 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (pool-4-thread-10) [57062bd1] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 0cfb1244-29ad-11e2-8844-52540035ed48 Type: VDS >2012-11-08 09:16:09,423 WARN [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (pool-4-thread-10) [5e1cad93] CanDoAction of action HandleVdsVersion failed. Reasons:VDS_CANNOT_CHECK_VERSION_HOST_NON_RESPONSIVE >2012-11-08 09:16:09,423 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (pool-4-thread-10) [5e1cad93] FINISH, ActivateVdsVDSCommand, log id: 506162a6 >2012-11-08 09:16:09,435 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (pool-4-thread-10) [5e1cad93] Lock freed to object EngineLock [exclusiveLocks= key: 0cfb1244-29ad-11e2-8844-52540035ed48 value: VDS >, sharedLocks= ] >2012-11-08 09:16:10,386 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-54) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:16:10,386 ERROR [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-54) VDS::handleNetworkException Server failed to respond, vds_id = 0cfb1244-29ad-11e2-8844-52540035ed48, vds_name = rhs-gp-srv12.lab.eng.blr.redhat.com, error = VDSNetworkException: >2012-11-08 09:16:10,393 INFO [org.ovirt.engine.core.bll.VdsEventListener] (pool-4-thread-10) ResourceManager::vdsNotResponding entered for Host 0cfb1244-29ad-11e2-8844-52540035ed48, 10.70.36.16 >2012-11-08 09:16:10,401 WARN [org.ovirt.engine.core.bll.VdsNotRespondingTreatmentCommand] (pool-4-thread-10) [370d38ec] CanDoAction of action VdsNotRespondingTreatment failed. Reasons:VDS_FENCING_DISABLED >2012-11-08 09:16:11,408 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (ajp-/127.0.0.1:8702-14) Lock Acquired to object EngineLock [exclusiveLocks= key: cfa543b0-29ac-11e2-b6f4-52540035ed48 value: VDS >, sharedLocks= ] >2012-11-08 09:16:11,424 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (pool-4-thread-10) [703451e3] Running command: ActivateVdsCommand internal: false. Entities affected : ID: cfa543b0-29ac-11e2-b6f4-52540035ed48 Type: VDS >2012-11-08 09:16:11,436 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-10) [703451e3] START, SetVdsStatusVDSCommand(HostName = rhs-gp-srv15.lab.eng.blr.redhat.com, HostId = cfa543b0-29ac-11e2-b6f4-52540035ed48, status=Unassigned, nonOperationalReason=NONE), log id: 606a0bdf >2012-11-08 09:16:11,438 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-10) [703451e3] FINISH, SetVdsStatusVDSCommand, log id: 606a0bdf >2012-11-08 09:16:11,441 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (pool-4-thread-10) [703451e3] START, ActivateVdsVDSCommand(HostName = rhs-gp-srv15.lab.eng.blr.redhat.com, HostId = cfa543b0-29ac-11e2-b6f4-52540035ed48), log id: 24f518ce >2012-11-08 09:16:11,446 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (pool-4-thread-10) [703451e3] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:16:11,446 ERROR [org.ovirt.engine.core.vdsbroker.VdsManager] (pool-4-thread-10) [703451e3] VDS::handleNetworkException Server failed to respond, vds_id = cfa543b0-29ac-11e2-b6f4-52540035ed48, vds_name = rhs-gp-srv15.lab.eng.blr.redhat.com, error = VDSNetworkException: >2012-11-08 09:16:11,457 INFO [org.ovirt.engine.core.bll.VdsEventListener] (pool-4-thread-45) [703451e3] ResourceManager::vdsNotResponding entered for Host cfa543b0-29ac-11e2-b6f4-52540035ed48, 10.70.36.19 >2012-11-08 09:16:11,461 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (pool-4-thread-10) [703451e3] ResourceManager::activateVds - failed to get VDS = cfa543b0-29ac-11e2-b6f4-52540035ed48 capabilities with error: VDSNetworkException: . >2012-11-08 09:16:11,461 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (pool-4-thread-10) [703451e3] ResourceManager::activateVds - failed to activate VDS = cfa543b0-29ac-11e2-b6f4-52540035ed48 >2012-11-08 09:16:11,466 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (pool-4-thread-10) [75d5677e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: cfa543b0-29ac-11e2-b6f4-52540035ed48 Type: VDS >2012-11-08 09:16:11,470 WARN [org.ovirt.engine.core.bll.VdsNotRespondingTreatmentCommand] (pool-4-thread-45) [16592738] CanDoAction of action VdsNotRespondingTreatment failed. Reasons:VDS_FENCING_DISABLED >2012-11-08 09:16:11,471 WARN [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (pool-4-thread-10) [7e12502a] CanDoAction of action HandleVdsVersion failed. Reasons:VDS_CANNOT_CHECK_VERSION_HOST_NON_RESPONSIVE >2012-11-08 09:16:11,471 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (pool-4-thread-10) [7e12502a] FINISH, ActivateVdsVDSCommand, log id: 24f518ce >2012-11-08 09:16:11,483 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (pool-4-thread-10) [7e12502a] Lock freed to object EngineLock [exclusiveLocks= key: cfa543b0-29ac-11e2-b6f4-52540035ed48 value: VDS >, sharedLocks= ] >2012-11-08 09:16:11,897 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-68) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:16:12,405 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-59) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:16:13,908 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-66) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:16:14,427 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-76) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:16:15,918 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-64) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:16:16,438 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-78) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:16:17,929 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-73) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:16:18,466 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-82) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:16:19,724 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-79) VM b5 3d766d10-58ab-4306-b5c4-7d109c7b6c46 moved from Up --> RebootInProgress >2012-11-08 09:16:19,733 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (QuartzScheduler_Worker-79) START, FullListVdsCommand( HostId = 96739be4-2990-11e2-8577-52540035ed48, vds=null, vmIds=[3d766d10-58ab-4306-b5c4-7d109c7b6c46]), log id: 4456e4c4 >2012-11-08 09:16:19,756 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (QuartzScheduler_Worker-79) FINISH, FullListVdsCommand, return: [Lorg.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcStruct;@24c522e9, log id: 4456e4c4 >2012-11-08 09:16:19,941 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-67) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:16:20,478 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-69) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:16:21,952 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-81) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:16:22,488 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-98) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:16:23,963 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-62) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:16:24,498 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-89) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:16:25,980 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-5) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:16:26,510 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-97) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:16:27,992 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-1) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:16:28,521 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-17) [36467d8f] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:16:30,002 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-12) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:16:30,532 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-4) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:16:32,013 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-18) [7ab5f38b] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:16:32,544 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-15) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:16:34,023 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-13) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:16:34,555 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-34) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:16:36,031 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-21) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:16:36,567 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-29) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:16:38,043 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-39) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:16:38,577 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-37) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:16:40,068 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-33) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:16:40,589 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-44) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:16:42,078 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-48) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:16:42,600 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-52) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:16:44,091 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-43) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:16:44,609 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-94) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:16:46,101 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-55) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:16:46,619 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-61) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:16:46,899 INFO [org.ovirt.engine.core.bll.MaintananceNumberOfVdssCommand] (pool-4-thread-10) [8a5697b] Running command: MaintananceNumberOfVdssCommand internal: false. Entities affected : ID: cfa543b0-29ac-11e2-b6f4-52540035ed48 Type: VDS >2012-11-08 09:16:46,918 INFO [org.ovirt.engine.core.bll.MaintananceVdsCommand] (pool-4-thread-10) [8a5697b] Running command: MaintananceVdsCommand internal: true. Entities affected : ID: cfa543b0-29ac-11e2-b6f4-52540035ed48 Type: VDS >2012-11-08 09:16:46,919 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-10) [8a5697b] START, SetVdsStatusVDSCommand(HostName = rhs-gp-srv15.lab.eng.blr.redhat.com, HostId = cfa543b0-29ac-11e2-b6f4-52540035ed48, status=Maintenance, nonOperationalReason=NONE), log id: a1b1c8e >2012-11-08 09:16:46,921 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-10) [8a5697b] FINISH, SetVdsStatusVDSCommand, log id: a1b1c8e >2012-11-08 09:16:48,629 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-46) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:16:49,636 INFO [org.ovirt.engine.core.bll.MaintananceNumberOfVdssCommand] (pool-4-thread-45) [67edb301] Running command: MaintananceNumberOfVdssCommand internal: false. Entities affected : ID: 0cfb1244-29ad-11e2-8844-52540035ed48 Type: VDS >2012-11-08 09:16:49,650 INFO [org.ovirt.engine.core.bll.MaintananceVdsCommand] (pool-4-thread-45) [67edb301] Running command: MaintananceVdsCommand internal: true. Entities affected : ID: 0cfb1244-29ad-11e2-8844-52540035ed48 Type: VDS >2012-11-08 09:16:49,652 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-45) [67edb301] START, SetVdsStatusVDSCommand(HostName = rhs-gp-srv12.lab.eng.blr.redhat.com, HostId = 0cfb1244-29ad-11e2-8844-52540035ed48, status=Maintenance, nonOperationalReason=NONE), log id: 7d871c4c >2012-11-08 09:16:49,653 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-45) [67edb301] FINISH, SetVdsStatusVDSCommand, log id: 7d871c4c >2012-11-08 09:16:52,507 INFO [org.ovirt.engine.core.bll.RemoveVdsCommand] (ajp-/127.0.0.1:8702-10) Lock Acquired to object EngineLock [exclusiveLocks= key: 0cfb1244-29ad-11e2-8844-52540035ed48 value: VDS >, sharedLocks= ] >2012-11-08 09:16:52,514 INFO [org.ovirt.engine.core.bll.RemoveVdsCommand] (pool-4-thread-45) [472da5] Running command: RemoveVdsCommand internal: false. Entities affected : ID: 0cfb1244-29ad-11e2-8844-52540035ed48 Type: VDS >2012-11-08 09:16:52,522 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (pool-4-thread-45) [472da5] START, RemoveVdsVDSCommand( HostId = 0cfb1244-29ad-11e2-8844-52540035ed48), log id: 25381649 >2012-11-08 09:16:52,522 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (pool-4-thread-45) [472da5] vdsManager::disposing >2012-11-08 09:16:52,522 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (pool-4-thread-45) [472da5] FINISH, RemoveVdsVDSCommand, log id: 25381649 >2012-11-08 09:16:52,522 INFO [org.ovirt.engine.core.bll.RemoveVdsCommand] (pool-4-thread-45) [472da5] Lock freed to object EngineLock [exclusiveLocks= key: 0cfb1244-29ad-11e2-8844-52540035ed48 value: VDS >, sharedLocks= ] >2012-11-08 09:16:55,392 INFO [org.ovirt.engine.core.bll.RemoveVdsCommand] (ajp-/127.0.0.1:8702-3) Lock Acquired to object EngineLock [exclusiveLocks= key: cfa543b0-29ac-11e2-b6f4-52540035ed48 value: VDS >, sharedLocks= ] >2012-11-08 09:16:55,399 INFO [org.ovirt.engine.core.bll.RemoveVdsCommand] (pool-4-thread-45) [5026093] Running command: RemoveVdsCommand internal: false. Entities affected : ID: cfa543b0-29ac-11e2-b6f4-52540035ed48 Type: VDS >2012-11-08 09:16:55,404 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (pool-4-thread-45) [5026093] START, RemoveVdsVDSCommand( HostId = cfa543b0-29ac-11e2-b6f4-52540035ed48), log id: 6231020 >2012-11-08 09:16:55,404 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (pool-4-thread-45) [5026093] vdsManager::disposing >2012-11-08 09:16:55,404 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (pool-4-thread-45) [5026093] FINISH, RemoveVdsVDSCommand, log id: 6231020 >2012-11-08 09:16:55,404 INFO [org.ovirt.engine.core.bll.RemoveVdsCommand] (pool-4-thread-45) [5026093] Lock freed to object EngineLock [exclusiveLocks= key: cfa543b0-29ac-11e2-b6f4-52540035ed48 value: VDS >, sharedLocks= ] >2012-11-08 09:17:20,172 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-29) VM b5 3d766d10-58ab-4306-b5c4-7d109c7b6c46 moved from RebootInProgress --> Up >2012-11-08 09:17:26,006 INFO [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (ajp-/127.0.0.1:8702-1) SSH execute 10.70.36.16 'IDFILE=/etc/vdsm/vdsm.id; if [ -r "${IDFILE}" ]; then cat "${IDFILE}"; else UUID="$(dmidecode -s system-uuid 2> /dev/null | sed -e 's/.*Not.*//' )"; if [ -z "${UUID}" ]; then UUID="$(uuidgen 2> /dev/null)" && mkdir -p "$(dirname "${IDFILE}")" && echo "${UUID}" > "${IDFILE}" && chmod 0644 "${IDFILE}"; fi; [ -n "${UUID}" ] && echo "${UUID}"; fi' >2012-11-08 09:17:26,140 INFO [org.ovirt.engine.core.bll.AddVdsCommand] (ajp-/127.0.0.1:8702-1) [5ac551b1] Running command: AddVdsCommand internal: false. Entities affected : ID: c04832b4-2976-11e2-b6fb-52540035ed48 Type: VdsGroups >2012-11-08 09:17:26,151 INFO [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp-/127.0.0.1:8702-1) [51464141] Running command: AddVdsSpmIdCommand internal: true. Entities affected : ID: 056a9890-29af-11e2-b5a8-52540035ed48 Type: VDS >2012-11-08 09:17:26,153 INFO [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp-/127.0.0.1:8702-1) [51464141] Lock freed to object EngineLock [exclusiveLocks= key: 1eaedbbb-1c0e-4028-9b0e-adac7e07d25f value: REGISTER_VDS >, sharedLocks= ] >2012-11-08 09:17:26,154 ERROR [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp-/127.0.0.1:8702-1) [51464141] Cannot get vdsManager for vdsid=056a9890-29af-11e2-b5a8-52540035ed48 >2012-11-08 09:17:26,154 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (ajp-/127.0.0.1:8702-1) [51464141] START, RemoveVdsVDSCommand(HostName = rhs-gp-srv12.lab.eng.blr.redhat.com, HostId = 056a9890-29af-11e2-b5a8-52540035ed48), log id: 2d67b332 >2012-11-08 09:17:26,154 ERROR [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp-/127.0.0.1:8702-1) [51464141] Cannot get vdsManager for vdsid=056a9890-29af-11e2-b5a8-52540035ed48 >2012-11-08 09:17:26,154 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (ajp-/127.0.0.1:8702-1) [51464141] FINISH, RemoveVdsVDSCommand, log id: 2d67b332 >2012-11-08 09:17:26,154 ERROR [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp-/127.0.0.1:8702-1) [51464141] Cannot get vdsManager for vdsid=056a9890-29af-11e2-b5a8-52540035ed48 >2012-11-08 09:17:26,155 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-1) [51464141] START, AddVdsVDSCommand(HostName = rhs-gp-srv12.lab.eng.blr.redhat.com, HostId = 056a9890-29af-11e2-b5a8-52540035ed48), log id: 6a96f45a >2012-11-08 09:17:26,155 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-1) [51464141] AddVds - entered , starting logic to add VDS 056a9890-29af-11e2-b5a8-52540035ed48 >2012-11-08 09:17:26,156 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-1) [51464141] AddVds - VDS 056a9890-29af-11e2-b5a8-52540035ed48 was added, will try to add it to the resource manager >2012-11-08 09:17:26,157 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ajp-/127.0.0.1:8702-1) [51464141] Eneterd VdsManager:constructor >2012-11-08 09:17:26,157 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ajp-/127.0.0.1:8702-1) [51464141] vdsBroker(10.70.36.16,54,321) >2012-11-08 09:17:26,157 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp-/127.0.0.1:8702-1) [51464141] ResourceManager::AddVds - VDS 056a9890-29af-11e2-b5a8-52540035ed48 was added to the Resource Manager >2012-11-08 09:17:26,157 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-1) [51464141] FINISH, AddVdsVDSCommand, log id: 6a96f45a >2012-11-08 09:17:26,169 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-4-thread-45) [1277110] Running command: InstallVdsCommand internal: true. Entities affected : ID: 056a9890-29af-11e2-b5a8-52540035ed48 Type: VDS >2012-11-08 09:17:26,170 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-4-thread-45) [1277110] Before Installation pool-4-thread-45 >2012-11-08 09:17:26,170 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-45) [1277110] Installation of 10.70.36.16. Executing installation stage. (Stage: Starting Host installation) >2012-11-08 09:17:26,170 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-45) [1277110] Installation of 10.70.36.16. Executing installation stage. (Stage: Connecting to Host) >2012-11-08 09:17:26,229 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-45) [1277110] Installation of 10.70.36.16. Received message: <BSTRAP component='RHEV_INSTALL' status='OK' message='Connected to Host 10.70.36.16 with SSH key fingerprint: d8:e6:4c:57:0c:54:e1:5a:31:d9:fa:80:44:2a:ac:7f'/>. FYI. (Stage: Connecting to Host) >2012-11-08 09:17:26,257 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-45) [1277110] Installation of 10.70.36.16. Successfully connected to server ssh. (Stage: Connecting to Host) >2012-11-08 09:17:26,257 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-45) [1277110] Installation of 10.70.36.16. Executing installation stage. (Stage: Get the unique vds id) >2012-11-08 09:17:26,257 INFO [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-4-thread-45) SSH execute 10.70.36.16 'IDFILE=/etc/vdsm/vdsm.id; if [ -r "${IDFILE}" ]; then cat "${IDFILE}"; else UUID="$(dmidecode -s system-uuid 2> /dev/null | sed -e 's/.*Not.*//' )"; if [ -z "${UUID}" ]; then UUID="$(uuidgen 2> /dev/null)" && mkdir -p "$(dirname "${IDFILE}")" && echo "${UUID}" > "${IDFILE}" && chmod 0644 "${IDFILE}"; fi; [ -n "${UUID}" ] && echo "${UUID}"; fi' >2012-11-08 09:17:26,372 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-97) Installation of 10.70.36.16. Received message: 00000000-0000-0000-0000-0025902F723C. FYI. (Stage: Get the unique vds id) >2012-11-08 09:17:26,375 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-97) Installation of 10.70.36.16. Assigning unique id 00000000-0000-0000-0000-0025902F723C to Host. (Stage: Get the unique vds id) >2012-11-08 09:17:26,378 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-45) [1277110] Installation of 10.70.36.16. Executing installation stage. (Stage: Upload Installation script to Host) >2012-11-08 09:17:26,379 INFO [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-4-thread-45) SSH execute 10.70.36.16 'cat > '/tmp/ovirt-id_rsa_34e19b8e-b1b5-4513-9d27-65e7d6b7ebfd'' >2012-11-08 09:17:26,464 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-45) [1277110] Installation of 10.70.36.16. Executing installation stage. (Stage: Running first installation script on Host) >2012-11-08 09:17:26,464 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-45) [1277110] Installation of 10.70.36.16. Sending SSH Command umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar -C "${MYTMP}" --no-same-permissions -o -x && "${MYTMP}"/setup -c 'ssl=true;management_port=54321' -O 'lab.eng.blr.redhat.com' -t 2012-11-08T14:17:26 -S /tmp/ovirt-id_rsa_34e19b8e-b1b5-4513-9d27-65e7d6b7ebfd -p 80 -b -B rhevm http://guido.lab.eng.blr.redhat.com:80/Components/vds/ http://guido.lab.eng.blr.redhat.com:80/Components/vds/ 10.70.36.16 34e19b8e-b1b5-4513-9d27-65e7d6b7ebfd False < /var/cache/ovirt-engine/vdsm-bootstrap-2.tar. (Stage: Running first installation script on Host) >2012-11-08 09:17:26,465 INFO [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-4-thread-45) SSH execute 10.70.36.16 'umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar -C "${MYTMP}" --no-same-permissions -o -x && "${MYTMP}"/setup -c 'ssl=true;management_port=54321' -O 'lab.eng.blr.redhat.com' -t 2012-11-08T14:17:26 -S /tmp/ovirt-id_rsa_34e19b8e-b1b5-4513-9d27-65e7d6b7ebfd -p 80 -b -B rhevm http://guido.lab.eng.blr.redhat.com:80/Components/vds/ http://guido.lab.eng.blr.redhat.com:80/Components/vds/ 10.70.36.16 34e19b8e-b1b5-4513-9d27-65e7d6b7ebfd False' >2012-11-08 09:17:26,560 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-97) Installation of 10.70.36.16. Received message: <BSTRAP component='INSTALLER' status='OK' message='Test platform succeeded'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:17:27,691 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-97) Installation of 10.70.36.16. Received message: <BSTRAP component='SET_SYSTEM_TIME' status='OK' message='setSystemTime ended successfully'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:17:28,159 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-49) Initializing Host: rhs-gp-srv12.lab.eng.blr.redhat.com >2012-11-08 09:17:30,560 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-97) Installation of 10.70.36.16. Received message: <BSTRAP component='RHN_REGISTRATION' status='OK' message='Host properly registered with RHN/Satellite.'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:17:30,735 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-97) Installation of 10.70.36.16. Received message: <BSTRAP component='VDSM_MAJOR_VER' status='OK' message='Available VDSM matches requirements'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:17:30,740 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-97) Installation of 10.70.36.16. Received message: <BSTRAP component='VT_SVM' status='OK' processor='Intel' message='Server supports virtualization'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:17:30,744 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-97) Installation of 10.70.36.16. Received message: <BSTRAP component='OS' status='OK' type='RHEL6' message='Supported platform version'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:17:30,748 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-97) Installation of 10.70.36.16. Received message: <BSTRAP component='KERNEL' status='OK' version='279' message='Supported kernel version: 279'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:17:30,856 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-97) Installation of 10.70.36.16. Received message: <BSTRAP component='REQ PACKAGES' status='OK' result='SDL.x86_64' message='SDL-1.2.14-3.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:17:30,873 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-97) Installation of 10.70.36.16. Received message: <BSTRAP component='REQ PACKAGES' status='OK' result='bridge-utils.x86_64' message='bridge-utils-1.2-9.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:17:30,889 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-97) Installation of 10.70.36.16. Received message: <BSTRAP component='REQ PACKAGES' status='OK' result='mesa-libGLU.x86_64' message='mesa-libGLU-7.11-5.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:17:30,909 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-97) Installation of 10.70.36.16. Received message: <BSTRAP component='REQ PACKAGES' status='OK' result='openssl.x86_64' message='openssl-1.0.0-25.el6_3.1.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:17:30,931 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-97) Installation of 10.70.36.16. Received message: <BSTRAP component='REQ PACKAGES' status='OK' result='m2crypto.x86_64' message='m2crypto-0.20.2-9.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:17:30,946 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-97) Installation of 10.70.36.16. Received message: <BSTRAP component='REQ PACKAGES' status='OK' result='rsync.x86_64' message='rsync-3.0.6-9.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:17:30,960 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-97) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='WARN' result='qemu-kvm' message='package qemu-kvm is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:17:30,973 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-97) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='WARN' result='qemu-kvm-tools' message='package qemu-kvm-tools is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:17:30,991 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-97) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='(~vdsm~, ~vdsm-cli~)' message='vdsm-4.9.6-41.0.el6_3.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:17:31,005 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-97) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='libjpeg' message='libjpeg-6b-46.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:17:31,019 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-97) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='spice-server' message='spice-server-0.10.1-10.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:17:31,034 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-97) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='pixman' message='pixman-0.18.4-1.el6_0.1.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:17:31,049 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-97) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='seabios' message='seabios-0.6.1.2-19.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:17:31,062 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-97) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='WARN' result='qemu-img' message='package qemu-img is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:17:31,080 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-97) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='fence-agents' message='fence-agents-3.1.5-17.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:17:31,098 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-97) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='libselinux-python' message='libselinux-python-2.0.94-5.3.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:17:31,111 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-97) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='sanlock' message='sanlock-2.3-4.el6_3.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:17:31,125 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-97) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='sanlock-python' message='sanlock-python-2.3-4.el6_3.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:17:48,601 INFO [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (ajp-/127.0.0.1:8702-10) SSH execute 10.70.36.19 'IDFILE=/etc/vdsm/vdsm.id; if [ -r "${IDFILE}" ]; then cat "${IDFILE}"; else UUID="$(dmidecode -s system-uuid 2> /dev/null | sed -e 's/.*Not.*//' )"; if [ -z "${UUID}" ]; then UUID="$(uuidgen 2> /dev/null)" && mkdir -p "$(dirname "${IDFILE}")" && echo "${UUID}" > "${IDFILE}" && chmod 0644 "${IDFILE}"; fi; [ -n "${UUID}" ] && echo "${UUID}"; fi' >2012-11-08 09:17:48,751 INFO [org.ovirt.engine.core.bll.AddVdsCommand] (ajp-/127.0.0.1:8702-10) [5ca384c6] Running command: AddVdsCommand internal: false. Entities affected : ID: c04832b4-2976-11e2-b6fb-52540035ed48 Type: VdsGroups >2012-11-08 09:17:48,766 INFO [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp-/127.0.0.1:8702-10) [3e61a574] Running command: AddVdsSpmIdCommand internal: true. Entities affected : ID: 12e4f376-29af-11e2-b479-52540035ed48 Type: VDS >2012-11-08 09:17:48,773 INFO [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp-/127.0.0.1:8702-10) [3e61a574] Lock freed to object EngineLock [exclusiveLocks= key: 1eaedbbb-1c0e-4028-9b0e-adac7e07d25f value: REGISTER_VDS >, sharedLocks= ] >2012-11-08 09:17:48,773 ERROR [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp-/127.0.0.1:8702-10) [3e61a574] Cannot get vdsManager for vdsid=12e4f376-29af-11e2-b479-52540035ed48 >2012-11-08 09:17:48,774 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (ajp-/127.0.0.1:8702-10) [3e61a574] START, RemoveVdsVDSCommand(HostName = rhs-gp-srv15.lab.eng.blr.redhat.com, HostId = 12e4f376-29af-11e2-b479-52540035ed48), log id: 7e889c07 >2012-11-08 09:17:48,774 ERROR [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp-/127.0.0.1:8702-10) [3e61a574] Cannot get vdsManager for vdsid=12e4f376-29af-11e2-b479-52540035ed48 >2012-11-08 09:17:48,774 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (ajp-/127.0.0.1:8702-10) [3e61a574] FINISH, RemoveVdsVDSCommand, log id: 7e889c07 >2012-11-08 09:17:48,774 ERROR [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp-/127.0.0.1:8702-10) [3e61a574] Cannot get vdsManager for vdsid=12e4f376-29af-11e2-b479-52540035ed48 >2012-11-08 09:17:48,774 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-10) [3e61a574] START, AddVdsVDSCommand(HostName = rhs-gp-srv15.lab.eng.blr.redhat.com, HostId = 12e4f376-29af-11e2-b479-52540035ed48), log id: 4ab5cd0c >2012-11-08 09:17:48,774 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-10) [3e61a574] AddVds - entered , starting logic to add VDS 12e4f376-29af-11e2-b479-52540035ed48 >2012-11-08 09:17:48,783 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-10) [3e61a574] AddVds - VDS 12e4f376-29af-11e2-b479-52540035ed48 was added, will try to add it to the resource manager >2012-11-08 09:17:48,783 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ajp-/127.0.0.1:8702-10) [3e61a574] Eneterd VdsManager:constructor >2012-11-08 09:17:48,784 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ajp-/127.0.0.1:8702-10) [3e61a574] vdsBroker(10.70.36.19,54,321) >2012-11-08 09:17:48,784 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp-/127.0.0.1:8702-10) [3e61a574] ResourceManager::AddVds - VDS 12e4f376-29af-11e2-b479-52540035ed48 was added to the Resource Manager >2012-11-08 09:17:48,784 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-10) [3e61a574] FINISH, AddVdsVDSCommand, log id: 4ab5cd0c >2012-11-08 09:17:48,800 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-4-thread-10) [106cea8a] Running command: InstallVdsCommand internal: true. Entities affected : ID: 12e4f376-29af-11e2-b479-52540035ed48 Type: VDS >2012-11-08 09:17:48,801 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-4-thread-10) [106cea8a] Before Installation pool-4-thread-10 >2012-11-08 09:17:48,801 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-10) [106cea8a] Installation of 10.70.36.19. Executing installation stage. (Stage: Starting Host installation) >2012-11-08 09:17:48,801 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-10) [106cea8a] Installation of 10.70.36.19. Executing installation stage. (Stage: Connecting to Host) >2012-11-08 09:17:48,867 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-10) [106cea8a] Installation of 10.70.36.19. Received message: <BSTRAP component='RHEV_INSTALL' status='OK' message='Connected to Host 10.70.36.19 with SSH key fingerprint: 74:1f:5c:d9:6d:8e:d7:72:33:74:37:01:2c:d8:1a:44'/>. FYI. (Stage: Connecting to Host) >2012-11-08 09:17:48,897 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-10) [106cea8a] Installation of 10.70.36.19. Successfully connected to server ssh. (Stage: Connecting to Host) >2012-11-08 09:17:48,897 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-10) [106cea8a] Installation of 10.70.36.19. Executing installation stage. (Stage: Get the unique vds id) >2012-11-08 09:17:48,897 INFO [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-4-thread-10) SSH execute 10.70.36.19 'IDFILE=/etc/vdsm/vdsm.id; if [ -r "${IDFILE}" ]; then cat "${IDFILE}"; else UUID="$(dmidecode -s system-uuid 2> /dev/null | sed -e 's/.*Not.*//' )"; if [ -z "${UUID}" ]; then UUID="$(uuidgen 2> /dev/null)" && mkdir -p "$(dirname "${IDFILE}")" && echo "${UUID}" > "${IDFILE}" && chmod 0644 "${IDFILE}"; fi; [ -n "${UUID}" ] && echo "${UUID}"; fi' >2012-11-08 09:17:49,005 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-105) Installation of 10.70.36.19. Received message: 00000000-0000-0000-0000-0025902F4FF8. FYI. (Stage: Get the unique vds id) >2012-11-08 09:17:49,006 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-105) Installation of 10.70.36.19. Assigning unique id 00000000-0000-0000-0000-0025902F4FF8 to Host. (Stage: Get the unique vds id) >2012-11-08 09:17:49,008 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-10) [106cea8a] Installation of 10.70.36.19. Executing installation stage. (Stage: Upload Installation script to Host) >2012-11-08 09:17:49,009 INFO [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-4-thread-10) SSH execute 10.70.36.19 'cat > '/tmp/ovirt-id_rsa_cff7580e-30a3-434d-a788-d1a7654b05a0'' >2012-11-08 09:17:49,092 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-10) [106cea8a] Installation of 10.70.36.19. Executing installation stage. (Stage: Running first installation script on Host) >2012-11-08 09:17:49,093 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-10) [106cea8a] Installation of 10.70.36.19. Sending SSH Command umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar -C "${MYTMP}" --no-same-permissions -o -x && "${MYTMP}"/setup -c 'ssl=true;management_port=54321' -O 'lab.eng.blr.redhat.com' -t 2012-11-08T14:17:49 -S /tmp/ovirt-id_rsa_cff7580e-30a3-434d-a788-d1a7654b05a0 -p 80 -b -B rhevm http://guido.lab.eng.blr.redhat.com:80/Components/vds/ http://guido.lab.eng.blr.redhat.com:80/Components/vds/ 10.70.36.19 cff7580e-30a3-434d-a788-d1a7654b05a0 False < /var/cache/ovirt-engine/vdsm-bootstrap-2.tar. (Stage: Running first installation script on Host) >2012-11-08 09:17:49,093 INFO [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-4-thread-10) SSH execute 10.70.36.19 'umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar -C "${MYTMP}" --no-same-permissions -o -x && "${MYTMP}"/setup -c 'ssl=true;management_port=54321' -O 'lab.eng.blr.redhat.com' -t 2012-11-08T14:17:49 -S /tmp/ovirt-id_rsa_cff7580e-30a3-434d-a788-d1a7654b05a0 -p 80 -b -B rhevm http://guido.lab.eng.blr.redhat.com:80/Components/vds/ http://guido.lab.eng.blr.redhat.com:80/Components/vds/ 10.70.36.19 cff7580e-30a3-434d-a788-d1a7654b05a0 False' >2012-11-08 09:17:49,201 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-105) Installation of 10.70.36.19. Received message: <BSTRAP component='INSTALLER' status='OK' message='Test platform succeeded'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:17:50,786 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-14) Initializing Host: rhs-gp-srv15.lab.eng.blr.redhat.com >2012-11-08 09:17:51,310 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-105) Installation of 10.70.36.19. Received message: <BSTRAP component='SET_SYSTEM_TIME' status='OK' message='setSystemTime ended successfully'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:17:54,993 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-105) Installation of 10.70.36.19. Received message: <BSTRAP component='RHN_REGISTRATION' status='OK' message='Host properly registered with RHN/Satellite.'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:17:55,169 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-105) Installation of 10.70.36.19. Received message: <BSTRAP component='VDSM_MAJOR_VER' status='OK' message='Available VDSM matches requirements'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:17:55,173 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-105) Installation of 10.70.36.19. Received message: <BSTRAP component='VT_SVM' status='OK' processor='Intel' message='Server supports virtualization'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:17:55,177 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-105) Installation of 10.70.36.19. Received message: <BSTRAP component='OS' status='OK' type='RHEL6' message='Supported platform version'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:17:55,181 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-105) Installation of 10.70.36.19. Received message: <BSTRAP component='KERNEL' status='OK' version='279' message='Supported kernel version: 279'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:17:55,282 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-105) Installation of 10.70.36.19. Received message: <BSTRAP component='REQ PACKAGES' status='OK' result='SDL.x86_64' message='SDL-1.2.14-3.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:17:55,305 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-105) Installation of 10.70.36.19. Received message: <BSTRAP component='REQ PACKAGES' status='OK' result='bridge-utils.x86_64' message='bridge-utils-1.2-9.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:17:55,321 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-105) Installation of 10.70.36.19. Received message: <BSTRAP component='REQ PACKAGES' status='OK' result='mesa-libGLU.x86_64' message='mesa-libGLU-7.11-5.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:17:55,338 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-105) Installation of 10.70.36.19. Received message: <BSTRAP component='REQ PACKAGES' status='OK' result='openssl.x86_64' message='openssl-1.0.0-25.el6_3.1.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:17:55,360 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-105) Installation of 10.70.36.19. Received message: <BSTRAP component='REQ PACKAGES' status='OK' result='m2crypto.x86_64' message='m2crypto-0.20.2-9.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:17:55,375 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-105) Installation of 10.70.36.19. Received message: <BSTRAP component='REQ PACKAGES' status='OK' result='rsync.x86_64' message='rsync-3.0.6-9.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:17:55,388 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-105) Installation of 10.70.36.19. Received message: <BSTRAP component='VDS PACKAGES' status='WARN' result='qemu-kvm' message='package qemu-kvm is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:17:55,401 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-105) Installation of 10.70.36.19. Received message: <BSTRAP component='VDS PACKAGES' status='WARN' result='qemu-kvm-tools' message='package qemu-kvm-tools is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:17:55,423 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-105) Installation of 10.70.36.19. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='(~vdsm~, ~vdsm-cli~)' message='vdsm-4.9.6-41.0.el6_3.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:17:55,438 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-105) Installation of 10.70.36.19. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='libjpeg' message='libjpeg-6b-46.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:17:55,452 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-105) Installation of 10.70.36.19. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='spice-server' message='spice-server-0.10.1-10.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:17:55,467 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-105) Installation of 10.70.36.19. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='pixman' message='pixman-0.18.4-1.el6_0.1.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:17:55,481 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-105) Installation of 10.70.36.19. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='seabios' message='seabios-0.6.1.2-19.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:17:55,494 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-105) Installation of 10.70.36.19. Received message: <BSTRAP component='VDS PACKAGES' status='WARN' result='qemu-img' message='package qemu-img is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:17:55,514 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-105) Installation of 10.70.36.19. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='fence-agents' message='fence-agents-3.1.5-17.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:17:55,532 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-105) Installation of 10.70.36.19. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='libselinux-python' message='libselinux-python-2.0.94-5.3.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:17:55,548 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-105) Installation of 10.70.36.19. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='sanlock' message='sanlock-2.3-4.el6_3.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:17:55,559 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-105) Installation of 10.70.36.19. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='sanlock-python' message='sanlock-python-2.3-4.el6_3.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:18:22,599 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-97) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='(~vdsm~, ~vdsm-cli~)' message='vdsm-4.9.6-41.0.el6_3.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:18:25,010 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-97) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='libjpeg' message='libjpeg-6b-46.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:18:27,462 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-97) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='spice-server' message='spice-server-0.10.1-10.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:18:29,875 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-97) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='pixman' message='pixman-0.18.4-1.el6_0.1.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:18:32,269 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-97) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='seabios' message='seabios-0.6.1.2-19.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:18:34,736 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-97) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='fence-agents' message='fence-agents-3.1.5-17.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:18:37,182 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-97) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='libselinux-python' message='libselinux-python-2.0.94-5.3.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:18:39,583 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-97) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='sanlock' message='sanlock-2.3-4.el6_3.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:18:42,038 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-97) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='sanlock-python' message='sanlock-python-2.3-4.el6_3.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:18:44,427 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-105) Installation of 10.70.36.19. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='(~vdsm~, ~vdsm-cli~)' message='vdsm-4.9.6-41.0.el6_3.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:18:44,487 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-97) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='qemu-img' message='package qemu-img is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:18:46,822 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-105) Installation of 10.70.36.19. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='libjpeg' message='libjpeg-6b-46.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:18:46,953 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-97) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='qemu-kvm-tools' message='package qemu-kvm-tools is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:18:49,274 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-105) Installation of 10.70.36.19. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='spice-server' message='spice-server-0.10.1-10.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:18:49,409 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-97) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='qemu-kvm' message='package qemu-kvm is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:18:49,413 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-97) Installation of 10.70.36.16. Received message: <BSTRAP component='CreateConf' status='OK' message='Basic configuration found, skipping this step'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:18:49,418 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-97) Installation of 10.70.36.16. Received message: <BSTRAP component='CheckLocalHostname' status='OK' message='Local hostname is correct.'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:18:49,472 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-97) Installation of 10.70.36.16. Received message: <BSTRAP component='SetNetworking' status='OK' message='Bridge management already exists. Skipping bridge creation.'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:18:49,587 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-97) Installation of 10.70.36.16. Received message: <BSTRAP component='SetSSHAccess' status='OK' message='SUCCESS'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:18:49,776 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-97) Installation of 10.70.36.16. Received message: <BSTRAP component='VerifyServices' status='OK' message='Needed services set'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:18:51,682 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-105) Installation of 10.70.36.19. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='pixman' message='pixman-0.18.4-1.el6_0.1.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:18:54,074 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-105) Installation of 10.70.36.19. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='seabios' message='seabios-0.6.1.2-19.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:18:56,750 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-97) Installation of 10.70.36.16. Received message: <BSTRAP component='Encryption setup' status='OK' message='Ended successfully'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:18:56,755 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-97) Installation of 10.70.36.16. Received message: <BSTRAP component='RHEV_INSTALL' status='OK'/>. Stage completed. (Stage: Running first installation script on Host) >2012-11-08 09:18:56,785 ERROR [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-45) [1277110] Installation of 10.70.36.16. Error: Repository JBOSS is listed more than once in the configuration >Repository qa-latest-5 is listed more than once in the configuration >Repository rhel-source is listed more than once in the configuration >Repository rhel-source-beta is listed more than once in the configuration >Repository rhs-client is listed more than once in the configuration >Repository JBOSS is listed more than once in the configuration >Repository qa-latest-5 is listed more than once in the configuration >Repository rhel-source is listed more than once in the configuration >Repository rhel-source-beta is listed more than once in the configuration >Repository rhs-client is listed more than once in the configuration >. (Stage: Downloading certificate request from Host) >2012-11-08 09:18:56,785 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-45) [1277110] VDS message: Repository JBOSS is listed more than once in the configuration >2012-11-08 09:18:56,785 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-45) [1277110] VDS message: Repository qa-latest-5 is listed more than once in the configuration >2012-11-08 09:18:56,785 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-45) [1277110] VDS message: Repository rhel-source is listed more than once in the configuration >2012-11-08 09:18:56,785 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-45) [1277110] VDS message: Repository rhel-source-beta is listed more than once in the configuration >2012-11-08 09:18:56,785 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-45) [1277110] VDS message: Repository rhs-client is listed more than once in the configuration >2012-11-08 09:18:56,785 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-45) [1277110] VDS message: Repository JBOSS is listed more than once in the configuration >2012-11-08 09:18:56,785 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-45) [1277110] VDS message: Repository qa-latest-5 is listed more than once in the configuration >2012-11-08 09:18:56,785 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-45) [1277110] VDS message: Repository rhel-source is listed more than once in the configuration >2012-11-08 09:18:56,785 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-45) [1277110] VDS message: Repository rhel-source-beta is listed more than once in the configuration >2012-11-08 09:18:56,785 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-45) [1277110] VDS message: Repository rhs-client is listed more than once in the configuration >2012-11-08 09:18:56,785 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-45) [1277110] Script ended, result is {1} >2012-11-08 09:18:56,785 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-45) [1277110] Installation of 10.70.36.16. Executing installation stage. (Stage: Downloading certificate request from Host) >2012-11-08 09:18:56,786 INFO [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-4-thread-45) SSH receive 10.70.36.16:'/tmp/cert_34e19b8e-b1b5-4513-9d27-65e7d6b7ebfd.req' '/etc/pki/ovirt-engine/requests/cert_34e19b8e-b1b5-4513-9d27-65e7d6b7ebfd.req') >2012-11-08 09:18:56,832 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-105) Installation of 10.70.36.19. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='fence-agents' message='fence-agents-3.1.5-17.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:18:56,857 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-45) [1277110] Installation of 10.70.36.16. successfully done sftp operation ( Stage: Downloading certificate request from Host) >2012-11-08 09:18:56,857 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-45) [1277110] DownloadCertificateRequest ended:true >2012-11-08 09:18:56,857 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-45) [1277110] Installation of 10.70.36.16. Executing installation stage. (Stage: Sign certificate request and generate certificate) >2012-11-08 09:18:57,863 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-45) [1277110] SignCertificateRequest ended:true >2012-11-08 09:18:57,863 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-45) [1277110] Installation of 10.70.36.16. Executing installation stage. (Stage: Upload signed sertificate to Host) >2012-11-08 09:18:57,885 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-45) [1277110] Installation of 10.70.36.16. successfully done sftp operation ( Stage: Upload signed sertificate to Host) >2012-11-08 09:18:57,885 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-45) [1277110] UploadSignedCertificate ended:true >2012-11-08 09:18:57,885 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-45) [1277110] Installation of 10.70.36.16. Executing installation stage. (Stage: Upload Cerficate Autority to Host) >2012-11-08 09:18:57,942 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-45) [1277110] Installation of 10.70.36.16. successfully done sftp operation ( Stage: Upload Cerficate Autority to Host) >2012-11-08 09:18:57,942 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-45) [1277110] Installation of 10.70.36.16. Executing installation stage. (Stage: Running second installation script on Host) >2012-11-08 09:18:57,942 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-45) [1277110] Installation of 10.70.36.16. Sending SSH Command umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar -C "${MYTMP}" --no-same-permissions -o -x && "${MYTMP}"/setup -c 'ssl=true;management_port=54321' -O 'lab.eng.blr.redhat.com' -t 2012-11-08T14:17:26 -S /tmp/ovirt-id_rsa_34e19b8e-b1b5-4513-9d27-65e7d6b7ebfd -p 80 -b -B rhevm http://guido.lab.eng.blr.redhat.com:80/Components/vds/ http://guido.lab.eng.blr.redhat.com:80/Components/vds/ 10.70.36.16 34e19b8e-b1b5-4513-9d27-65e7d6b7ebfd True < /var/cache/ovirt-engine/vdsm-bootstrap-2.tar. (Stage: Running second installation script on Host) >2012-11-08 09:18:57,942 INFO [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-4-thread-45) SSH execute 10.70.36.16 'umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar -C "${MYTMP}" --no-same-permissions -o -x && "${MYTMP}"/setup -c 'ssl=true;management_port=54321' -O 'lab.eng.blr.redhat.com' -t 2012-11-08T14:17:26 -S /tmp/ovirt-id_rsa_34e19b8e-b1b5-4513-9d27-65e7d6b7ebfd -p 80 -b -B rhevm http://guido.lab.eng.blr.redhat.com:80/Components/vds/ http://guido.lab.eng.blr.redhat.com:80/Components/vds/ 10.70.36.16 34e19b8e-b1b5-4513-9d27-65e7d6b7ebfd True' >2012-11-08 09:18:58,031 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-97) Installation of 10.70.36.16. Received message: <BSTRAP component='INSTALLER' status='OK' message='Test platform succeeded'/>. FYI. (Stage: Running second installation script on Host) >2012-11-08 09:18:58,386 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-97) Installation of 10.70.36.16. Received message: <BSTRAP component='instCert' status='OK'/>. FYI. (Stage: Running second installation script on Host) >2012-11-08 09:18:58,390 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-97) Installation of 10.70.36.16. Received message: <BSTRAP component='CoreDump' status='OK'/>. FYI. (Stage: Running second installation script on Host) >2012-11-08 09:18:58,394 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-97) Installation of 10.70.36.16. Received message: <BSTRAP component='cleanAll' status='OK'/>. FYI. (Stage: Running second installation script on Host) >2012-11-08 09:18:58,397 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-97) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS Configuration' status='OK'/>. FYI. (Stage: Running second installation script on Host) >2012-11-08 09:18:58,399 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-97) Installation of 10.70.36.16. Received message: <BSTRAP component='RHEV_INSTALL' status='OK'/>. Stage completed. (Stage: Running second installation script on Host) >2012-11-08 09:18:58,409 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-97) Installation of 10.70.36.16. Received message: <BSTRAP component='Reboot' status='OK' message='Rebooting machine' />. FYI. (Stage: Host installation complete) >2012-11-08 09:18:58,431 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-45) [1277110] Script ended, result is {1} >2012-11-08 09:18:58,431 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-4-thread-45) [1277110] After Installation pool-4-thread-45 >2012-11-08 09:18:58,435 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-45) [1277110] START, SetVdsStatusVDSCommand(HostName = rhs-gp-srv12.lab.eng.blr.redhat.com, HostId = 056a9890-29af-11e2-b5a8-52540035ed48, status=Reboot, nonOperationalReason=NONE), log id: 267c15e1 >2012-11-08 09:18:58,442 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-45) [1277110] FINISH, SetVdsStatusVDSCommand, log id: 267c15e1 >2012-11-08 09:18:58,442 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-4-thread-50) [1277110] Waiting 300 seconds, for server to finish reboot process. >2012-11-08 09:18:59,287 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-105) Installation of 10.70.36.19. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='libselinux-python' message='libselinux-python-2.0.94-5.3.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:19:02,004 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-105) Installation of 10.70.36.19. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='sanlock' message='sanlock-2.3-4.el6_3.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:19:04,447 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-105) Installation of 10.70.36.19. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='sanlock-python' message='sanlock-python-2.3-4.el6_3.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:19:06,912 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-105) Installation of 10.70.36.19. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='qemu-img' message='package qemu-img is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:19:09,388 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-105) Installation of 10.70.36.19. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='qemu-kvm-tools' message='package qemu-kvm-tools is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:19:11,876 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-105) Installation of 10.70.36.19. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='qemu-kvm' message='package qemu-kvm is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:19:11,882 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-105) Installation of 10.70.36.19. Received message: <BSTRAP component='CreateConf' status='OK' message='Basic configuration found, skipping this step'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:19:11,888 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-105) Installation of 10.70.36.19. Received message: <BSTRAP component='CheckLocalHostname' status='OK' message='Local hostname is correct.'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:19:11,941 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-105) Installation of 10.70.36.19. Received message: <BSTRAP component='SetNetworking' status='OK' message='Bridge management already exists. Skipping bridge creation.'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:19:12,055 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-105) Installation of 10.70.36.19. Received message: <BSTRAP component='SetSSHAccess' status='OK' message='SUCCESS'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:19:12,242 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-105) Installation of 10.70.36.19. Received message: <BSTRAP component='VerifyServices' status='OK' message='Needed services set'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:19:24,333 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-105) Installation of 10.70.36.19. Received message: <BSTRAP component='Encryption setup' status='OK' message='Ended successfully'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:19:24,338 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-105) Installation of 10.70.36.19. Received message: <BSTRAP component='RHEV_INSTALL' status='OK'/>. Stage completed. (Stage: Running first installation script on Host) >2012-11-08 09:19:24,367 ERROR [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-10) [106cea8a] Installation of 10.70.36.19. Error: Repository JBOSS is listed more than once in the configuration >Repository qa-latest-5 is listed more than once in the configuration >Repository rhel-source is listed more than once in the configuration >Repository rhel-source-beta is listed more than once in the configuration >Repository rhs-client is listed more than once in the configuration >Repository JBOSS is listed more than once in the configuration >Repository qa-latest-5 is listed more than once in the configuration >Repository rhel-source is listed more than once in the configuration >Repository rhel-source-beta is listed more than once in the configuration >Repository rhs-client is listed more than once in the configuration >. (Stage: Downloading certificate request from Host) >2012-11-08 09:19:24,368 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-10) [106cea8a] VDS message: Repository JBOSS is listed more than once in the configuration >2012-11-08 09:19:24,368 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-10) [106cea8a] VDS message: Repository qa-latest-5 is listed more than once in the configuration >2012-11-08 09:19:24,368 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-10) [106cea8a] VDS message: Repository rhel-source is listed more than once in the configuration >2012-11-08 09:19:24,368 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-10) [106cea8a] VDS message: Repository rhel-source-beta is listed more than once in the configuration >2012-11-08 09:19:24,368 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-10) [106cea8a] VDS message: Repository rhs-client is listed more than once in the configuration >2012-11-08 09:19:24,368 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-10) [106cea8a] VDS message: Repository JBOSS is listed more than once in the configuration >2012-11-08 09:19:24,368 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-10) [106cea8a] VDS message: Repository qa-latest-5 is listed more than once in the configuration >2012-11-08 09:19:24,368 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-10) [106cea8a] VDS message: Repository rhel-source is listed more than once in the configuration >2012-11-08 09:19:24,368 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-10) [106cea8a] VDS message: Repository rhel-source-beta is listed more than once in the configuration >2012-11-08 09:19:24,368 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-10) [106cea8a] VDS message: Repository rhs-client is listed more than once in the configuration >2012-11-08 09:19:24,368 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-10) [106cea8a] Script ended, result is {1} >2012-11-08 09:19:24,368 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-10) [106cea8a] Installation of 10.70.36.19. Executing installation stage. (Stage: Downloading certificate request from Host) >2012-11-08 09:19:24,368 INFO [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-4-thread-10) SSH receive 10.70.36.19:'/tmp/cert_cff7580e-30a3-434d-a788-d1a7654b05a0.req' '/etc/pki/ovirt-engine/requests/cert_cff7580e-30a3-434d-a788-d1a7654b05a0.req') >2012-11-08 09:19:24,441 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-10) [106cea8a] Installation of 10.70.36.19. successfully done sftp operation ( Stage: Downloading certificate request from Host) >2012-11-08 09:19:24,441 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-10) [106cea8a] DownloadCertificateRequest ended:true >2012-11-08 09:19:24,441 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-10) [106cea8a] Installation of 10.70.36.19. Executing installation stage. (Stage: Sign certificate request and generate certificate) >2012-11-08 09:19:25,443 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-10) [106cea8a] SignCertificateRequest ended:true >2012-11-08 09:19:25,443 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-10) [106cea8a] Installation of 10.70.36.19. Executing installation stage. (Stage: Upload signed sertificate to Host) >2012-11-08 09:19:25,467 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-10) [106cea8a] Installation of 10.70.36.19. successfully done sftp operation ( Stage: Upload signed sertificate to Host) >2012-11-08 09:19:25,467 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-10) [106cea8a] UploadSignedCertificate ended:true >2012-11-08 09:19:25,467 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-10) [106cea8a] Installation of 10.70.36.19. Executing installation stage. (Stage: Upload Cerficate Autority to Host) >2012-11-08 09:19:25,522 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-10) [106cea8a] Installation of 10.70.36.19. successfully done sftp operation ( Stage: Upload Cerficate Autority to Host) >2012-11-08 09:19:25,523 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-10) [106cea8a] Installation of 10.70.36.19. Executing installation stage. (Stage: Running second installation script on Host) >2012-11-08 09:19:25,523 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-10) [106cea8a] Installation of 10.70.36.19. Sending SSH Command umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar -C "${MYTMP}" --no-same-permissions -o -x && "${MYTMP}"/setup -c 'ssl=true;management_port=54321' -O 'lab.eng.blr.redhat.com' -t 2012-11-08T14:17:49 -S /tmp/ovirt-id_rsa_cff7580e-30a3-434d-a788-d1a7654b05a0 -p 80 -b -B rhevm http://guido.lab.eng.blr.redhat.com:80/Components/vds/ http://guido.lab.eng.blr.redhat.com:80/Components/vds/ 10.70.36.19 cff7580e-30a3-434d-a788-d1a7654b05a0 True < /var/cache/ovirt-engine/vdsm-bootstrap-2.tar. (Stage: Running second installation script on Host) >2012-11-08 09:19:25,523 INFO [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-4-thread-10) SSH execute 10.70.36.19 'umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar -C "${MYTMP}" --no-same-permissions -o -x && "${MYTMP}"/setup -c 'ssl=true;management_port=54321' -O 'lab.eng.blr.redhat.com' -t 2012-11-08T14:17:49 -S /tmp/ovirt-id_rsa_cff7580e-30a3-434d-a788-d1a7654b05a0 -p 80 -b -B rhevm http://guido.lab.eng.blr.redhat.com:80/Components/vds/ http://guido.lab.eng.blr.redhat.com:80/Components/vds/ 10.70.36.19 cff7580e-30a3-434d-a788-d1a7654b05a0 True' >2012-11-08 09:19:25,617 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-105) Installation of 10.70.36.19. Received message: <BSTRAP component='INSTALLER' status='OK' message='Test platform succeeded'/>. FYI. (Stage: Running second installation script on Host) >2012-11-08 09:19:25,976 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-105) Installation of 10.70.36.19. Received message: <BSTRAP component='instCert' status='OK'/>. FYI. (Stage: Running second installation script on Host) >2012-11-08 09:19:25,981 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-105) Installation of 10.70.36.19. Received message: <BSTRAP component='CoreDump' status='OK'/>. FYI. (Stage: Running second installation script on Host) >2012-11-08 09:19:25,985 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-105) Installation of 10.70.36.19. Received message: <BSTRAP component='cleanAll' status='OK'/>. FYI. (Stage: Running second installation script on Host) >2012-11-08 09:19:25,989 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-105) Installation of 10.70.36.19. Received message: <BSTRAP component='VDS Configuration' status='OK'/>. FYI. (Stage: Running second installation script on Host) >2012-11-08 09:19:25,993 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-105) Installation of 10.70.36.19. Received message: <BSTRAP component='RHEV_INSTALL' status='OK'/>. Stage completed. (Stage: Running second installation script on Host) >2012-11-08 09:19:26,003 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-105) Installation of 10.70.36.19. Received message: <BSTRAP component='Reboot' status='OK' message='Rebooting machine' />. FYI. (Stage: Host installation complete) >2012-11-08 09:19:26,022 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-10) [106cea8a] Script ended, result is {1} >2012-11-08 09:19:26,022 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-4-thread-10) [106cea8a] After Installation pool-4-thread-10 >2012-11-08 09:19:26,023 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-10) [106cea8a] START, SetVdsStatusVDSCommand(HostName = rhs-gp-srv15.lab.eng.blr.redhat.com, HostId = 12e4f376-29af-11e2-b479-52540035ed48, status=Reboot, nonOperationalReason=NONE), log id: 6a964721 >2012-11-08 09:19:26,030 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-10) [106cea8a] FINISH, SetVdsStatusVDSCommand, log id: 6a964721 >2012-11-08 09:19:26,030 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-4-thread-47) [106cea8a] Waiting 300 seconds, for server to finish reboot process. >2012-11-08 09:19:55,943 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-65) DbUserCacheManager::refreshAllUserData() - entered >2012-11-08 09:19:58,284 INFO [org.ovirt.engine.core.bll.LogoutUserCommand] (ajp-/127.0.0.1:8702-7) [308c2018] Running command: LogoutUserCommand internal: false. >2012-11-08 09:20:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-69) Autorecovering hosts is disabled, skipping >2012-11-08 09:20:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-69) Autorecovering storage domains is disabled, skipping >2012-11-08 09:22:21,259 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) domain a02e96be-cf4c-4d0f-8368-fd9d4aa659bf:REBDOMAIN in problem. vds: rhs-gp-srv11.lab.eng.blr.redhat.com >2012-11-08 09:23:20,519 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-70) [c5e7fc0] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:23:20,548 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-70) [c5e7fc0] IrsBroker::Failed::GetStoragePoolInfoVDS due to: ConnectException: Connection refused >2012-11-08 09:23:21,016 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-78) ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = 96739be4-2990-11e2-8577-52540035ed48 : rhs-gp-srv11.lab.eng.blr.redhat.com, VDS Network Error, continuing. >VDSNetworkException: >2012-11-08 09:23:23,105 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-77) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@30679e39 >2012-11-08 09:23:23,105 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-77) HostName = rhs-gp-srv11.lab.eng.blr.redhat.com >2012-11-08 09:23:23,105 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-77) Command GetCapabilitiesVDS execution failed. Error: VDSRecoveringException: Recovering from crash or Initializing >2012-11-08 09:23:23,126 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-77) ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = 96739be4-2990-11e2-8577-52540035ed48 : rhs-gp-srv11.lab.eng.blr.redhat.com, error = Recovering from crash or Initializing, continuing. >2012-11-08 09:23:25,696 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-67) Cancelling the recovery from crash timer for VDS 96739be4-2990-11e2-8577-52540035ed48 because vds started initializing >2012-11-08 09:23:25,698 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-67) [777467dc] Running command: InitVdsOnUpCommand internal: true. >2012-11-08 09:23:25,760 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-67) [175b4033] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:23:25,774 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (QuartzScheduler_Worker-67) [175b4033] START, ConnectStorageServerVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, storageType = POSIXFS, connectionList = [{ id: 0d9a1047-add1-41b9-911c-bf284b6fbfb2, connection: rhs-client37.lab.eng.blr.redhat.com:dist-rep, iqn: null, vfsType: glusterfs, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 301f13d5-f9ce-47d1-a2ef-ec72ec607bac, connection: rhs-client36.lab.eng.blr.redhat.com:distribute, iqn: null, vfsType: glusterfs, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: f30a8ea >2012-11-08 09:23:25,797 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (QuartzScheduler_Worker-67) [175b4033] FINISH, ConnectStorageServerVDSCommand, return: {0d9a1047-add1-41b9-911c-bf284b6fbfb2=0, 301f13d5-f9ce-47d1-a2ef-ec72ec607bac=0}, log id: f30a8ea >2012-11-08 09:23:25,797 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-67) [175b4033] Host rhs-gp-srv11.lab.eng.blr.redhat.com storage connection was succeeded >2012-11-08 09:23:25,813 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (QuartzScheduler_Worker-67) [175b4033] START, ConnectStorageServerVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, storageType = NFS, connectionList = [{ id: 658918ca-5dd4-4f39-9d39-f4a117794418, connection: guido.lab.eng.blr.redhat.com:/iso2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 18094235 >2012-11-08 09:23:25,839 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (QuartzScheduler_Worker-67) [175b4033] FINISH, ConnectStorageServerVDSCommand, return: {658918ca-5dd4-4f39-9d39-f4a117794418=0}, log id: 18094235 >2012-11-08 09:23:25,839 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-67) [175b4033] Host rhs-gp-srv11.lab.eng.blr.redhat.com storage connection was succeeded >2012-11-08 09:23:25,857 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-67) [175b4033] START, ConnectStoragePoolVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, vds_spm_id = 1, masterDomainId = a02e96be-cf4c-4d0f-8368-fd9d4aa659bf, masterVersion = 1), log id: 17f72190 >2012-11-08 09:23:30,749 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-67) [175b4033] FINISH, ConnectStoragePoolVDSCommand, log id: 17f72190 >2012-11-08 09:23:30,827 ERROR [org.ovirt.engine.core.bll.FencingExecutor] (QuartzScheduler_Worker-67) [175b4033] Failed to run Power Management command on Host rhs-gp-srv11.lab.eng.blr.redhat.com, no running proxy Host was found. >2012-11-08 09:23:30,865 INFO [org.ovirt.engine.core.bll.MultipleActionsRunner] (pool-4-thread-49) [175b4033] MultipleActionsRunner of type MigrateVmToServer invoked with no actions >2012-11-08 09:23:30,901 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-67) [191f572d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 96739be4-2990-11e2-8577-52540035ed48 Type: VDS >2012-11-08 09:23:30,915 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-67) [4bc2377f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 96739be4-2990-11e2-8577-52540035ed48 Type: VDS >2012-11-08 09:23:35,666 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-93) [23ef92ac] Waiting to Host rhs-gp-srv11.lab.eng.blr.redhat.com to finish initialization for 5 Sec. >2012-11-08 09:23:35,711 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-93) [23ef92ac] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:23:35,737 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-93) [23ef92ac] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:23:35,740 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-93) [23ef92ac] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:23:35,754 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-93) [23ef92ac] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 77603f22 >2012-11-08 09:23:35,776 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-93) [23ef92ac] spmStart polling started: taskId = cc5af4f8-d4c3-4bd4-a1c7-ffb127a00d01 >2012-11-08 09:23:39,176 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (QuartzScheduler_Worker-25) START, FullListVdsCommand( HostId = 96739be4-2990-11e2-8577-52540035ed48, vds=null, vmIds=[67bd2d56-8e0b-4328-bcfd-ea7f4758c885, 68655de5-7bd9-4bdc-8628-3b2069ec312f]), log id: 6ce4aef2 >2012-11-08 09:23:39,200 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (QuartzScheduler_Worker-25) FINISH, FullListVdsCommand, return: [Lorg.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcStruct;@1999c9b8, log id: 6ce4aef2 >2012-11-08 09:23:57,468 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-93) [23ef92ac] Failed in HSMGetTaskStatusVDS method >2012-11-08 09:23:57,468 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-93) [23ef92ac] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:23:57,469 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-93) [23ef92ac] spmStart polling ended: taskId = cc5af4f8-d4c3-4bd4-a1c7-ffb127a00d01 task status = finished >2012-11-08 09:23:57,469 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-93) [23ef92ac] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:23:57,494 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-93) [23ef92ac] spmStart polling ended, spm status: Free >2012-11-08 09:23:57,506 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-93) [23ef92ac] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=cc5af4f8-d4c3-4bd4-a1c7-ffb127a00d01), log id: 21405524 >2012-11-08 09:23:57,524 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-93) [23ef92ac] FINISH, HSMClearTaskVDSCommand, log id: 21405524 >2012-11-08 09:23:57,524 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-93) [23ef92ac] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@695bf8a, log id: 77603f22 >2012-11-08 09:23:57,526 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-93) [51520066] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:23:57,549 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-93) [51520066] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:23:58,445 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-50) [1277110] START, SetVdsStatusVDSCommand(HostName = rhs-gp-srv12.lab.eng.blr.redhat.com, HostId = 056a9890-29af-11e2-b5a8-52540035ed48, status=NonResponsive, nonOperationalReason=NONE), log id: 2fffc909 >2012-11-08 09:23:58,456 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-50) [1277110] FINISH, SetVdsStatusVDSCommand, log id: 2fffc909 >2012-11-08 09:23:58,530 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-73) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:23:59,904 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-80) Domain a02e96be-cf4c-4d0f-8368-fd9d4aa659bf:REBDOMAIN recovered from problem. vds: rhs-gp-srv11.lab.eng.blr.redhat.com >2012-11-08 09:23:59,904 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-80) Domain a02e96be-cf4c-4d0f-8368-fd9d4aa659bf:REBDOMAIN has recovered from problem. No active host in the DC is reporting it as problematic, so clearing the domain recovery timer. >2012-11-08 09:24:00,537 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-79) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:24:02,544 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-87) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:24:04,555 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-96) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:24:06,565 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-3) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:24:07,605 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-99) hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:24:07,632 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-99) SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:24:07,635 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-99) starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:24:07,652 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-99) START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6e401ec7 >2012-11-08 09:24:07,681 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-99) spmStart polling started: taskId = 761f7383-a34d-4ddb-9eba-4dac53865bcd >2012-11-08 09:24:08,576 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-1) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:24:08,717 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-99) Failed in HSMGetTaskStatusVDS method >2012-11-08 09:24:08,718 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-99) Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:24:08,718 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-99) spmStart polling ended: taskId = 761f7383-a34d-4ddb-9eba-4dac53865bcd task status = finished >2012-11-08 09:24:08,718 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-99) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:24:08,751 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-99) spmStart polling ended, spm status: Free >2012-11-08 09:24:08,769 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-99) START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=761f7383-a34d-4ddb-9eba-4dac53865bcd), log id: 5cc993f4 >2012-11-08 09:24:08,794 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-99) FINISH, HSMClearTaskVDSCommand, log id: 5cc993f4 >2012-11-08 09:24:08,794 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-99) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3610a7c3, log id: 6e401ec7 >2012-11-08 09:24:08,796 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-99) [7f760eef] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:24:08,823 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-99) [7f760eef] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:24:10,588 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-51) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:24:12,599 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-24) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:24:14,609 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-27) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:24:16,619 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-30) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:24:18,645 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-39) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:24:18,880 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-19) hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:24:18,909 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-19) SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:24:18,913 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-19) starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:24:18,930 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-19) START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6055e526 >2012-11-08 09:24:18,959 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-19) spmStart polling started: taskId = b68ba087-78c4-4fae-8a68-d2a5ccd25ed6 >2012-11-08 09:24:20,004 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-19) Failed in HSMGetTaskStatusVDS method >2012-11-08 09:24:20,004 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-19) Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:24:20,004 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-19) spmStart polling ended: taskId = b68ba087-78c4-4fae-8a68-d2a5ccd25ed6 task status = finished >2012-11-08 09:24:20,004 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-19) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:24:20,033 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-19) spmStart polling ended, spm status: Free >2012-11-08 09:24:20,050 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-19) START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=b68ba087-78c4-4fae-8a68-d2a5ccd25ed6), log id: 1b67afd >2012-11-08 09:24:20,075 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-19) FINISH, HSMClearTaskVDSCommand, log id: 1b67afd >2012-11-08 09:24:20,075 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-19) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@734478eb, log id: 6055e526 >2012-11-08 09:24:20,077 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-19) [145ec76c] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:24:20,098 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-19) [145ec76c] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:24:20,656 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-37) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:24:22,667 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-49) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:24:24,677 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-43) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:24:26,032 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-47) [106cea8a] START, SetVdsStatusVDSCommand(HostName = rhs-gp-srv15.lab.eng.blr.redhat.com, HostId = 12e4f376-29af-11e2-b479-52540035ed48, status=NonResponsive, nonOperationalReason=NONE), log id: 3777c93 >2012-11-08 09:24:26,037 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-47) [106cea8a] FINISH, SetVdsStatusVDSCommand, log id: 3777c93 >2012-11-08 09:24:26,689 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-68) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:24:27,141 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-61) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:24:28,700 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-56) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:24:29,149 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-46) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:24:30,151 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:24:30,178 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:24:30,182 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:24:30,199 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-65) START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7589d61b >2012-11-08 09:24:30,221 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-65) spmStart polling started: taskId = 69f7d34c-f921-4e4a-aece-d1a4f7b3d050 >2012-11-08 09:24:30,708 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-64) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:24:31,159 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-71) [3e904d2f] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:24:31,251 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-65) Failed in HSMGetTaskStatusVDS method >2012-11-08 09:24:31,251 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-65) Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:24:31,252 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-65) spmStart polling ended: taskId = 69f7d34c-f921-4e4a-aece-d1a4f7b3d050 task status = finished >2012-11-08 09:24:31,252 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-65) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:24:31,280 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-65) spmStart polling ended, spm status: Free >2012-11-08 09:24:31,298 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-65) START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=69f7d34c-f921-4e4a-aece-d1a4f7b3d050), log id: 4ceb5efe >2012-11-08 09:24:31,317 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-65) FINISH, HSMClearTaskVDSCommand, log id: 4ceb5efe >2012-11-08 09:24:31,317 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-65) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7feef0da, log id: 7589d61b >2012-11-08 09:24:31,319 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-65) [2006b9d4] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:24:31,341 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) [2006b9d4] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:24:32,718 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-35) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:24:33,170 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-83) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:24:34,729 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-80) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:24:35,182 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-90) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:24:36,741 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-86) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:24:37,193 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-87) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:24:38,750 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-98) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:24:39,203 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-96) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:24:40,760 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-95) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:24:41,213 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-100) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:24:41,409 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:24:41,437 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:24:41,440 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:24:41,458 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-97) START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 20abc15a >2012-11-08 09:24:41,479 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-97) spmStart polling started: taskId = 26424844-7320-49a8-abbb-c053ea35dba5 >2012-11-08 09:24:42,511 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-97) Failed in HSMGetTaskStatusVDS method >2012-11-08 09:24:42,511 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-97) Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:24:42,511 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-97) spmStart polling ended: taskId = 26424844-7320-49a8-abbb-c053ea35dba5 task status = finished >2012-11-08 09:24:42,511 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-97) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:24:42,538 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-97) spmStart polling ended, spm status: Free >2012-11-08 09:24:42,557 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-97) START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=26424844-7320-49a8-abbb-c053ea35dba5), log id: 4607e5e1 >2012-11-08 09:24:42,574 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-97) FINISH, HSMClearTaskVDSCommand, log id: 4607e5e1 >2012-11-08 09:24:42,574 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-97) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@30853ab2, log id: 20abc15a >2012-11-08 09:24:42,576 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-97) [7ad09500] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:24:42,598 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [7ad09500] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:24:42,768 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-1) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:24:43,224 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-8) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:24:44,777 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-2) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:24:45,232 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-7) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:24:46,788 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-4) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:24:47,238 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-24) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:24:48,798 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-15) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:24:49,246 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-25) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:24:50,282 INFO [org.ovirt.engine.core.bll.MaintananceNumberOfVdssCommand] (pool-4-thread-50) [733bbec8] Running command: MaintananceNumberOfVdssCommand internal: false. Entities affected : ID: 056a9890-29af-11e2-b5a8-52540035ed48 Type: VDS >2012-11-08 09:24:50,317 INFO [org.ovirt.engine.core.bll.MaintananceVdsCommand] (pool-4-thread-50) [733bbec8] Running command: MaintananceVdsCommand internal: true. Entities affected : ID: 056a9890-29af-11e2-b5a8-52540035ed48 Type: VDS >2012-11-08 09:24:50,318 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-50) [733bbec8] START, SetVdsStatusVDSCommand(HostName = rhs-gp-srv12.lab.eng.blr.redhat.com, HostId = 056a9890-29af-11e2-b5a8-52540035ed48, status=Maintenance, nonOperationalReason=NONE), log id: 7f7752f3 >2012-11-08 09:24:50,320 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-50) [733bbec8] FINISH, SetVdsStatusVDSCommand, log id: 7f7752f3 >2012-11-08 09:24:51,256 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-30) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:24:52,661 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-29) hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:24:52,689 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-29) SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:24:52,692 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-29) starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:24:52,709 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-29) START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 55652a80 >2012-11-08 09:24:52,738 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-29) spmStart polling started: taskId = 73bc72e3-6d2d-49ee-b5f0-492045183dd8 >2012-11-08 09:24:53,268 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-36) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:24:53,778 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-29) Failed in HSMGetTaskStatusVDS method >2012-11-08 09:24:53,778 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-29) Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:24:53,778 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-29) spmStart polling ended: taskId = 73bc72e3-6d2d-49ee-b5f0-492045183dd8 task status = finished >2012-11-08 09:24:53,778 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-29) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:24:53,808 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-29) spmStart polling ended, spm status: Free >2012-11-08 09:24:53,821 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-29) START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=73bc72e3-6d2d-49ee-b5f0-492045183dd8), log id: 23357bb6 >2012-11-08 09:24:53,841 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-29) FINISH, HSMClearTaskVDSCommand, log id: 23357bb6 >2012-11-08 09:24:53,841 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-29) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4f1278f, log id: 55652a80 >2012-11-08 09:24:53,842 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-29) [745c1fd1] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:24:53,864 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-29) [745c1fd1] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:24:55,279 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-37) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:24:55,768 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (pool-4-thread-50) START, IsValidVDSCommand( storagePoolId = f4c98a0b-0578-485f-90f4-743df3d05d7e, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 15a6925a >2012-11-08 09:24:55,769 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (pool-4-thread-50) FINISH, IsValidVDSCommand, return: true, log id: 15a6925a >2012-11-08 09:24:55,770 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = f4c98a0b-0578-485f-90f4-743df3d05d7e, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 43c054ae >2012-11-08 09:24:55,771 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = rhs-arch-srv2.lab.eng.blr.redhat.com, HostId = cf017816-29a2-11e2-949e-52540035ed48, storagePoolId=f4c98a0b-0578-485f-90f4-743df3d05d7e), log id: b9f0753 >2012-11-08 09:24:55,801 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [RHEL6.3-20120613.2-Server-x86_64-DVD1.iso], log id: b9f0753 >2012-11-08 09:24:55,801 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [RHEL6.3-20120613.2-Server-x86_64-DVD1.iso], log id: 43c054ae >2012-11-08 09:24:55,804 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 7a268ac7-857a-4c53-966d-247deb509161. >2012-11-08 09:24:57,291 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-52) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:24:59,302 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-50) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:25:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-57) Autorecovering hosts is disabled, skipping >2012-11-08 09:25:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-57) Autorecovering storage domains is disabled, skipping >2012-11-08 09:25:01,313 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-59) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:25:01,580 INFO [org.ovirt.engine.core.bll.MaintananceNumberOfVdssCommand] (pool-4-thread-50) [491d3078] Running command: MaintananceNumberOfVdssCommand internal: false. Entities affected : ID: 12e4f376-29af-11e2-b479-52540035ed48 Type: VDS >2012-11-08 09:25:01,592 INFO [org.ovirt.engine.core.bll.MaintananceVdsCommand] (pool-4-thread-50) [491d3078] Running command: MaintananceVdsCommand internal: true. Entities affected : ID: 12e4f376-29af-11e2-b479-52540035ed48 Type: VDS >2012-11-08 09:25:01,594 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-50) [491d3078] START, SetVdsStatusVDSCommand(HostName = rhs-gp-srv15.lab.eng.blr.redhat.com, HostId = 12e4f376-29af-11e2-b479-52540035ed48, status=Maintenance, nonOperationalReason=NONE), log id: 4c872b30 >2012-11-08 09:25:01,595 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-50) [491d3078] FINISH, SetVdsStatusVDSCommand, log id: 4c872b30 >2012-11-08 09:25:03,922 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [3e904d2f] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:25:03,943 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [3e904d2f] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:25:03,947 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [3e904d2f] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:25:03,964 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-71) [3e904d2f] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 58714d0f >2012-11-08 09:25:03,985 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-71) [3e904d2f] spmStart polling started: taskId = 6e306c49-2059-4a69-a569-0625812c3f64 >2012-11-08 09:25:05,018 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-71) [3e904d2f] Failed in HSMGetTaskStatusVDS method >2012-11-08 09:25:05,018 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-71) [3e904d2f] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:25:05,018 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-71) [3e904d2f] spmStart polling ended: taskId = 6e306c49-2059-4a69-a569-0625812c3f64 task status = finished >2012-11-08 09:25:05,018 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-71) [3e904d2f] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:25:05,048 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-71) [3e904d2f] spmStart polling ended, spm status: Free >2012-11-08 09:25:05,066 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-71) [3e904d2f] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=6e306c49-2059-4a69-a569-0625812c3f64), log id: 4fbf5d3b >2012-11-08 09:25:05,084 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-71) [3e904d2f] FINISH, HSMClearTaskVDSCommand, log id: 4fbf5d3b >2012-11-08 09:25:05,084 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-71) [3e904d2f] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1c81f564, log id: 58714d0f >2012-11-08 09:25:05,086 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-71) [181586db] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:25:05,106 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [181586db] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:25:15,155 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:25:15,185 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:25:15,188 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:25:15,206 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 290ef212 >2012-11-08 09:25:15,228 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) spmStart polling started: taskId = 69a5cf88-657c-439f-a92d-750d97293640 >2012-11-08 09:25:16,259 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-14) Failed in HSMGetTaskStatusVDS method >2012-11-08 09:25:16,259 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-14) Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:25:16,259 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) spmStart polling ended: taskId = 69a5cf88-657c-439f-a92d-750d97293640 task status = finished >2012-11-08 09:25:16,260 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:25:16,282 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) spmStart polling ended, spm status: Free >2012-11-08 09:25:16,293 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-14) START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=69a5cf88-657c-439f-a92d-750d97293640), log id: 58dc518c >2012-11-08 09:25:16,315 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-14) FINISH, HSMClearTaskVDSCommand, log id: 58dc518c >2012-11-08 09:25:16,315 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@686debe7, log id: 290ef212 >2012-11-08 09:25:16,317 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-14) [89ac041] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:25:16,340 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [89ac041] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:25:26,407 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-36) hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:25:26,431 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-36) SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:25:26,434 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-36) starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:25:26,449 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-36) START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4d55ad71 >2012-11-08 09:25:26,478 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-36) spmStart polling started: taskId = fb965ba9-8b01-4eb5-a85c-bfd57f94c98e >2012-11-08 09:25:27,107 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp-/127.0.0.1:8702-2) [6524645d] Running command: UpdateVdsCommand internal: false. Entities affected : ID: 056a9890-29af-11e2-b5a8-52540035ed48 Type: VDS >2012-11-08 09:25:27,115 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp-/127.0.0.1:8702-2) [6524645d] START, SetVdsStatusVDSCommand(HostName = rhs-gp-srv12.lab.eng.blr.redhat.com, HostId = 056a9890-29af-11e2-b5a8-52540035ed48, status=Installing, nonOperationalReason=NONE), log id: 64bd25f7 >2012-11-08 09:25:27,117 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp-/127.0.0.1:8702-2) [6524645d] FINISH, SetVdsStatusVDSCommand, log id: 64bd25f7 >2012-11-08 09:25:27,120 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-4-thread-47) [488d048b] Running command: InstallVdsCommand internal: true. Entities affected : ID: 056a9890-29af-11e2-b5a8-52540035ed48 Type: VDS >2012-11-08 09:25:27,121 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-4-thread-47) [488d048b] Before Installation pool-4-thread-47 >2012-11-08 09:25:27,121 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-47) [488d048b] Installation of 10.70.36.16. Executing installation stage. (Stage: Starting Host installation) >2012-11-08 09:25:27,122 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-47) [488d048b] Installation of 10.70.36.16. Executing installation stage. (Stage: Connecting to Host) >2012-11-08 09:25:27,189 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-47) [488d048b] Installation of 10.70.36.16. Received message: <BSTRAP component='RHEV_INSTALL' status='OK' message='Connected to Host 10.70.36.16 with SSH key fingerprint: d8:e6:4c:57:0c:54:e1:5a:31:d9:fa:80:44:2a:ac:7f'/>. FYI. (Stage: Connecting to Host) >2012-11-08 09:25:27,219 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-47) [488d048b] Installation of 10.70.36.16. Successfully connected to server ssh. (Stage: Connecting to Host) >2012-11-08 09:25:27,219 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-47) [488d048b] Installation of 10.70.36.16. Executing installation stage. (Stage: Get the unique vds id) >2012-11-08 09:25:27,219 INFO [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-4-thread-47) SSH execute 10.70.36.16 'IDFILE=/etc/vdsm/vdsm.id; if [ -r "${IDFILE}" ]; then cat "${IDFILE}"; else UUID="$(dmidecode -s system-uuid 2> /dev/null | sed -e 's/.*Not.*//' )"; if [ -z "${UUID}" ]; then UUID="$(uuidgen 2> /dev/null)" && mkdir -p "$(dirname "${IDFILE}")" && echo "${UUID}" > "${IDFILE}" && chmod 0644 "${IDFILE}"; fi; [ -n "${UUID}" ] && echo "${UUID}"; fi' >2012-11-08 09:25:27,339 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-106) Installation of 10.70.36.16. Received message: 00000000-0000-0000-0000-0025902F723C. FYI. (Stage: Get the unique vds id) >2012-11-08 09:25:27,349 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-106) Installation of 10.70.36.16. Assigning unique id 00000000-0000-0000-0000-0025902F723C to Host. (Stage: Get the unique vds id) >2012-11-08 09:25:27,352 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-47) [488d048b] Installation of 10.70.36.16. Executing installation stage. (Stage: Upload Installation script to Host) >2012-11-08 09:25:27,352 INFO [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-4-thread-47) SSH execute 10.70.36.16 'cat > '/tmp/ovirt-id_rsa_4f881ef1-f92f-411f-881d-f008966c8670'' >2012-11-08 09:25:27,438 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-47) [488d048b] Installation of 10.70.36.16. Executing installation stage. (Stage: Running first installation script on Host) >2012-11-08 09:25:27,438 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-47) [488d048b] Installation of 10.70.36.16. Sending SSH Command umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar -C "${MYTMP}" --no-same-permissions -o -x && "${MYTMP}"/setup -c 'ssl=true;management_port=54321' -O 'lab.eng.blr.redhat.com' -t 2012-11-08T14:25:27 -S /tmp/ovirt-id_rsa_4f881ef1-f92f-411f-881d-f008966c8670 -p 80 -b -B rhevm http://guido.lab.eng.blr.redhat.com:80/Components/vds/ http://guido.lab.eng.blr.redhat.com:80/Components/vds/ 10.70.36.16 4f881ef1-f92f-411f-881d-f008966c8670 False < /var/cache/ovirt-engine/vdsm-bootstrap-2.tar. (Stage: Running first installation script on Host) >2012-11-08 09:25:27,438 INFO [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-4-thread-47) SSH execute 10.70.36.16 'umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar -C "${MYTMP}" --no-same-permissions -o -x && "${MYTMP}"/setup -c 'ssl=true;management_port=54321' -O 'lab.eng.blr.redhat.com' -t 2012-11-08T14:25:27 -S /tmp/ovirt-id_rsa_4f881ef1-f92f-411f-881d-f008966c8670 -p 80 -b -B rhevm http://guido.lab.eng.blr.redhat.com:80/Components/vds/ http://guido.lab.eng.blr.redhat.com:80/Components/vds/ 10.70.36.16 4f881ef1-f92f-411f-881d-f008966c8670 False' >2012-11-08 09:25:27,510 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-36) Failed in HSMGetTaskStatusVDS method >2012-11-08 09:25:27,510 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-36) Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:25:27,510 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-36) spmStart polling ended: taskId = fb965ba9-8b01-4eb5-a85c-bfd57f94c98e task status = finished >2012-11-08 09:25:27,510 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-36) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:25:27,537 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-36) spmStart polling ended, spm status: Free >2012-11-08 09:25:27,545 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-106) Installation of 10.70.36.16. Received message: <BSTRAP component='INSTALLER' status='OK' message='Test platform succeeded'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:25:27,556 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-36) START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=fb965ba9-8b01-4eb5-a85c-bfd57f94c98e), log id: 79e29866 >2012-11-08 09:25:27,581 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-36) FINISH, HSMClearTaskVDSCommand, log id: 79e29866 >2012-11-08 09:25:27,581 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-36) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@b838f0e, log id: 4d55ad71 >2012-11-08 09:25:27,583 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-36) [6a24e609] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:25:27,603 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-36) [6a24e609] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:25:28,688 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-106) Installation of 10.70.36.16. Received message: <BSTRAP component='SET_SYSTEM_TIME' status='OK' message='setSystemTime ended successfully'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:25:31,853 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-106) Installation of 10.70.36.16. Received message: <BSTRAP component='RHN_REGISTRATION' status='OK' message='Host properly registered with RHN/Satellite.'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:25:32,028 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-106) Installation of 10.70.36.16. Received message: <BSTRAP component='VDSM_MAJOR_VER' status='OK' message='Available VDSM matches requirements'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:25:32,032 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-106) Installation of 10.70.36.16. Received message: <BSTRAP component='VT_SVM' status='OK' processor='Intel' message='Server supports virtualization'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:25:32,036 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-106) Installation of 10.70.36.16. Received message: <BSTRAP component='OS' status='OK' type='RHEL6' message='Supported platform version'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:25:32,040 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-106) Installation of 10.70.36.16. Received message: <BSTRAP component='KERNEL' status='OK' version='279' message='Supported kernel version: 279'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:25:32,145 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-106) Installation of 10.70.36.16. Received message: <BSTRAP component='REQ PACKAGES' status='OK' result='SDL.x86_64' message='SDL-1.2.14-3.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:25:32,166 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-106) Installation of 10.70.36.16. Received message: <BSTRAP component='REQ PACKAGES' status='OK' result='bridge-utils.x86_64' message='bridge-utils-1.2-9.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:25:32,182 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-106) Installation of 10.70.36.16. Received message: <BSTRAP component='REQ PACKAGES' status='OK' result='mesa-libGLU.x86_64' message='mesa-libGLU-7.11-5.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:25:32,202 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-106) Installation of 10.70.36.16. Received message: <BSTRAP component='REQ PACKAGES' status='OK' result='openssl.x86_64' message='openssl-1.0.0-25.el6_3.1.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:25:32,225 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-106) Installation of 10.70.36.16. Received message: <BSTRAP component='REQ PACKAGES' status='OK' result='m2crypto.x86_64' message='m2crypto-0.20.2-9.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:25:32,240 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-106) Installation of 10.70.36.16. Received message: <BSTRAP component='REQ PACKAGES' status='OK' result='rsync.x86_64' message='rsync-3.0.6-9.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:25:32,253 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-106) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='WARN' result='qemu-kvm' message='package qemu-kvm is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:25:32,266 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-106) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='WARN' result='qemu-kvm-tools' message='package qemu-kvm-tools is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:25:32,284 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-106) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='(~vdsm~, ~vdsm-cli~)' message='vdsm-4.9.6-41.0.el6_3.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:25:32,298 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-106) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='libjpeg' message='libjpeg-6b-46.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:25:32,313 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-106) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='spice-server' message='spice-server-0.10.1-10.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:25:32,327 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-106) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='pixman' message='pixman-0.18.4-1.el6_0.1.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:25:32,342 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-106) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='seabios' message='seabios-0.6.1.2-19.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:25:32,355 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-106) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='WARN' result='qemu-img' message='package qemu-img is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:25:32,374 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-106) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='fence-agents' message='fence-agents-3.1.5-17.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:25:32,391 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-106) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='libselinux-python' message='libselinux-python-2.0.94-5.3.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:25:32,405 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-106) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='sanlock' message='sanlock-2.3-4.el6_3.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:25:32,418 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-106) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='sanlock-python' message='sanlock-python-2.3-4.el6_3.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:25:37,659 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-76) hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:25:37,687 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-76) SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:25:37,689 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-76) starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:25:37,706 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-76) START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6c5bf1a5 >2012-11-08 09:25:37,735 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-76) spmStart polling started: taskId = 1f7112e1-f268-4601-8e87-6a476e8db0c8 >2012-11-08 09:25:38,781 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-76) Failed in HSMGetTaskStatusVDS method >2012-11-08 09:25:38,781 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-76) Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:25:38,781 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-76) spmStart polling ended: taskId = 1f7112e1-f268-4601-8e87-6a476e8db0c8 task status = finished >2012-11-08 09:25:38,781 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-76) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:25:38,808 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-76) spmStart polling ended, spm status: Free >2012-11-08 09:25:38,821 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-76) START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=1f7112e1-f268-4601-8e87-6a476e8db0c8), log id: 4486167a >2012-11-08 09:25:38,839 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-76) FINISH, HSMClearTaskVDSCommand, log id: 4486167a >2012-11-08 09:25:38,839 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-76) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2fb935c7, log id: 6c5bf1a5 >2012-11-08 09:25:38,840 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-76) [2879b44b] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:25:38,860 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-76) [2879b44b] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:25:41,869 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp-/127.0.0.1:8702-11) [1713bbd0] Running command: UpdateVdsCommand internal: false. Entities affected : ID: 12e4f376-29af-11e2-b479-52540035ed48 Type: VDS >2012-11-08 09:25:41,875 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp-/127.0.0.1:8702-11) [1713bbd0] START, SetVdsStatusVDSCommand(HostName = rhs-gp-srv15.lab.eng.blr.redhat.com, HostId = 12e4f376-29af-11e2-b479-52540035ed48, status=Installing, nonOperationalReason=NONE), log id: d01408c >2012-11-08 09:25:41,877 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp-/127.0.0.1:8702-11) [1713bbd0] FINISH, SetVdsStatusVDSCommand, log id: d01408c >2012-11-08 09:25:41,880 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-4-thread-45) [28e3a2cf] Running command: InstallVdsCommand internal: true. Entities affected : ID: 12e4f376-29af-11e2-b479-52540035ed48 Type: VDS >2012-11-08 09:25:41,881 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-4-thread-45) [28e3a2cf] Before Installation pool-4-thread-45 >2012-11-08 09:25:41,881 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-45) [28e3a2cf] Installation of 10.70.36.19. Executing installation stage. (Stage: Starting Host installation) >2012-11-08 09:25:41,881 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-45) [28e3a2cf] Installation of 10.70.36.19. Executing installation stage. (Stage: Connecting to Host) >2012-11-08 09:25:41,952 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-45) [28e3a2cf] Installation of 10.70.36.19. Received message: <BSTRAP component='RHEV_INSTALL' status='OK' message='Connected to Host 10.70.36.19 with SSH key fingerprint: 74:1f:5c:d9:6d:8e:d7:72:33:74:37:01:2c:d8:1a:44'/>. FYI. (Stage: Connecting to Host) >2012-11-08 09:25:41,985 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-45) [28e3a2cf] Installation of 10.70.36.19. Successfully connected to server ssh. (Stage: Connecting to Host) >2012-11-08 09:25:41,985 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-45) [28e3a2cf] Installation of 10.70.36.19. Executing installation stage. (Stage: Get the unique vds id) >2012-11-08 09:25:41,985 INFO [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-4-thread-45) SSH execute 10.70.36.19 'IDFILE=/etc/vdsm/vdsm.id; if [ -r "${IDFILE}" ]; then cat "${IDFILE}"; else UUID="$(dmidecode -s system-uuid 2> /dev/null | sed -e 's/.*Not.*//' )"; if [ -z "${UUID}" ]; then UUID="$(uuidgen 2> /dev/null)" && mkdir -p "$(dirname "${IDFILE}")" && echo "${UUID}" > "${IDFILE}" && chmod 0644 "${IDFILE}"; fi; [ -n "${UUID}" ] && echo "${UUID}"; fi' >2012-11-08 09:25:42,105 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-110) Installation of 10.70.36.19. Received message: 00000000-0000-0000-0000-0025902F4FF8. FYI. (Stage: Get the unique vds id) >2012-11-08 09:25:42,116 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-110) Installation of 10.70.36.19. Assigning unique id 00000000-0000-0000-0000-0025902F4FF8 to Host. (Stage: Get the unique vds id) >2012-11-08 09:25:42,119 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-45) [28e3a2cf] Installation of 10.70.36.19. Executing installation stage. (Stage: Upload Installation script to Host) >2012-11-08 09:25:42,119 INFO [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-4-thread-45) SSH execute 10.70.36.19 'cat > '/tmp/ovirt-id_rsa_606a7cd9-9517-4da1-9a6d-c6b19fb18b2e'' >2012-11-08 09:25:42,204 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-45) [28e3a2cf] Installation of 10.70.36.19. Executing installation stage. (Stage: Running first installation script on Host) >2012-11-08 09:25:42,204 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-45) [28e3a2cf] Installation of 10.70.36.19. Sending SSH Command umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar -C "${MYTMP}" --no-same-permissions -o -x && "${MYTMP}"/setup -c 'ssl=true;management_port=54321' -O 'lab.eng.blr.redhat.com' -t 2012-11-08T14:25:42 -S /tmp/ovirt-id_rsa_606a7cd9-9517-4da1-9a6d-c6b19fb18b2e -p 80 -b -B rhevm http://guido.lab.eng.blr.redhat.com:80/Components/vds/ http://guido.lab.eng.blr.redhat.com:80/Components/vds/ 10.70.36.19 606a7cd9-9517-4da1-9a6d-c6b19fb18b2e False < /var/cache/ovirt-engine/vdsm-bootstrap-2.tar. (Stage: Running first installation script on Host) >2012-11-08 09:25:42,204 INFO [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-4-thread-45) SSH execute 10.70.36.19 'umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar -C "${MYTMP}" --no-same-permissions -o -x && "${MYTMP}"/setup -c 'ssl=true;management_port=54321' -O 'lab.eng.blr.redhat.com' -t 2012-11-08T14:25:42 -S /tmp/ovirt-id_rsa_606a7cd9-9517-4da1-9a6d-c6b19fb18b2e -p 80 -b -B rhevm http://guido.lab.eng.blr.redhat.com:80/Components/vds/ http://guido.lab.eng.blr.redhat.com:80/Components/vds/ 10.70.36.19 606a7cd9-9517-4da1-9a6d-c6b19fb18b2e False' >2012-11-08 09:25:42,311 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-110) Installation of 10.70.36.19. Received message: <BSTRAP component='INSTALLER' status='OK' message='Test platform succeeded'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:25:44,432 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-110) Installation of 10.70.36.19. Received message: <BSTRAP component='SET_SYSTEM_TIME' status='OK' message='setSystemTime ended successfully'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:25:47,282 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-110) Installation of 10.70.36.19. Received message: <BSTRAP component='RHN_REGISTRATION' status='OK' message='Host properly registered with RHN/Satellite.'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:25:47,452 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-110) Installation of 10.70.36.19. Received message: <BSTRAP component='VDSM_MAJOR_VER' status='OK' message='Available VDSM matches requirements'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:25:47,456 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-110) Installation of 10.70.36.19. Received message: <BSTRAP component='VT_SVM' status='OK' processor='Intel' message='Server supports virtualization'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:25:47,459 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-110) Installation of 10.70.36.19. Received message: <BSTRAP component='OS' status='OK' type='RHEL6' message='Supported platform version'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:25:47,462 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-110) Installation of 10.70.36.19. Received message: <BSTRAP component='KERNEL' status='OK' version='279' message='Supported kernel version: 279'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:25:47,564 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-110) Installation of 10.70.36.19. Received message: <BSTRAP component='REQ PACKAGES' status='OK' result='SDL.x86_64' message='SDL-1.2.14-3.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:25:47,582 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-110) Installation of 10.70.36.19. Received message: <BSTRAP component='REQ PACKAGES' status='OK' result='bridge-utils.x86_64' message='bridge-utils-1.2-9.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:25:47,598 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-110) Installation of 10.70.36.19. Received message: <BSTRAP component='REQ PACKAGES' status='OK' result='mesa-libGLU.x86_64' message='mesa-libGLU-7.11-5.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:25:47,614 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-110) Installation of 10.70.36.19. Received message: <BSTRAP component='REQ PACKAGES' status='OK' result='openssl.x86_64' message='openssl-1.0.0-25.el6_3.1.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:25:47,636 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-110) Installation of 10.70.36.19. Received message: <BSTRAP component='REQ PACKAGES' status='OK' result='m2crypto.x86_64' message='m2crypto-0.20.2-9.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:25:47,651 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-110) Installation of 10.70.36.19. Received message: <BSTRAP component='REQ PACKAGES' status='OK' result='rsync.x86_64' message='rsync-3.0.6-9.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:25:47,664 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-110) Installation of 10.70.36.19. Received message: <BSTRAP component='VDS PACKAGES' status='WARN' result='qemu-kvm' message='package qemu-kvm is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:25:47,677 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-110) Installation of 10.70.36.19. Received message: <BSTRAP component='VDS PACKAGES' status='WARN' result='qemu-kvm-tools' message='package qemu-kvm-tools is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:25:47,700 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-110) Installation of 10.70.36.19. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='(~vdsm~, ~vdsm-cli~)' message='vdsm-4.9.6-41.0.el6_3.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:25:47,714 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-110) Installation of 10.70.36.19. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='libjpeg' message='libjpeg-6b-46.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:25:47,729 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-110) Installation of 10.70.36.19. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='spice-server' message='spice-server-0.10.1-10.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:25:47,743 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-110) Installation of 10.70.36.19. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='pixman' message='pixman-0.18.4-1.el6_0.1.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:25:47,758 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-110) Installation of 10.70.36.19. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='seabios' message='seabios-0.6.1.2-19.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:25:47,771 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-110) Installation of 10.70.36.19. Received message: <BSTRAP component='VDS PACKAGES' status='WARN' result='qemu-img' message='package qemu-img is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:25:47,791 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-110) Installation of 10.70.36.19. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='fence-agents' message='fence-agents-3.1.5-17.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:25:47,809 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-110) Installation of 10.70.36.19. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='libselinux-python' message='libselinux-python-2.0.94-5.3.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:25:47,822 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-110) Installation of 10.70.36.19. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='sanlock' message='sanlock-2.3-4.el6_3.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:25:47,836 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-110) Installation of 10.70.36.19. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='sanlock-python' message='sanlock-python-2.3-4.el6_3.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:25:48,926 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-67) [4bc2377f] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:25:48,955 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-67) [4bc2377f] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:25:48,958 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-67) [4bc2377f] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:25:48,975 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-67) [4bc2377f] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6cb505db >2012-11-08 09:25:49,001 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-67) [4bc2377f] spmStart polling started: taskId = 2f46e696-758a-4503-af9d-87729a2fc77c >2012-11-08 09:25:50,038 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-67) [4bc2377f] Failed in HSMGetTaskStatusVDS method >2012-11-08 09:25:50,038 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-67) [4bc2377f] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:25:50,038 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-67) [4bc2377f] spmStart polling ended: taskId = 2f46e696-758a-4503-af9d-87729a2fc77c task status = finished >2012-11-08 09:25:50,038 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-67) [4bc2377f] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:25:50,067 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-67) [4bc2377f] spmStart polling ended, spm status: Free >2012-11-08 09:25:50,080 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-67) [4bc2377f] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=2f46e696-758a-4503-af9d-87729a2fc77c), log id: 22132c6d >2012-11-08 09:25:50,103 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-67) [4bc2377f] FINISH, HSMClearTaskVDSCommand, log id: 22132c6d >2012-11-08 09:25:50,103 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-67) [4bc2377f] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@18c9b995, log id: 6cb505db >2012-11-08 09:25:50,105 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-67) [63b6f1e9] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:25:50,130 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-67) [63b6f1e9] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:26:00,190 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:26:00,216 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:26:00,220 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:26:00,236 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-41) START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 697f4c45 >2012-11-08 09:26:00,265 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-41) spmStart polling started: taskId = 63e7c271-6be7-43fd-94d3-b74f76952d17 >2012-11-08 09:26:01,297 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-41) Failed in HSMGetTaskStatusVDS method >2012-11-08 09:26:01,297 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-41) Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:26:01,297 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-41) spmStart polling ended: taskId = 63e7c271-6be7-43fd-94d3-b74f76952d17 task status = finished >2012-11-08 09:26:01,297 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-41) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:26:01,327 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-41) spmStart polling ended, spm status: Free >2012-11-08 09:26:01,344 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-41) START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=63e7c271-6be7-43fd-94d3-b74f76952d17), log id: 66374289 >2012-11-08 09:26:01,372 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-41) FINISH, HSMClearTaskVDSCommand, log id: 66374289 >2012-11-08 09:26:01,372 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-41) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2b69111, log id: 697f4c45 >2012-11-08 09:26:01,374 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-41) [3f4dc79d] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:26:01,398 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) [3f4dc79d] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:26:11,449 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:26:11,476 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:26:11,478 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:26:11,495 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-63) START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: e1feb9 >2012-11-08 09:26:11,519 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-63) spmStart polling started: taskId = 8fe4ef44-3d2e-421b-9c66-774d2a273750 >2012-11-08 09:26:12,553 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-63) Failed in HSMGetTaskStatusVDS method >2012-11-08 09:26:12,553 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-63) Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:26:12,553 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-63) spmStart polling ended: taskId = 8fe4ef44-3d2e-421b-9c66-774d2a273750 task status = finished >2012-11-08 09:26:12,553 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-63) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:26:12,582 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-63) spmStart polling ended, spm status: Free >2012-11-08 09:26:12,597 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-63) START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=8fe4ef44-3d2e-421b-9c66-774d2a273750), log id: 59a19840 >2012-11-08 09:26:12,620 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-63) FINISH, HSMClearTaskVDSCommand, log id: 59a19840 >2012-11-08 09:26:12,620 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-63) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@12bfb9, log id: e1feb9 >2012-11-08 09:26:12,623 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-63) [4ee90ab8] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:26:12,641 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) [4ee90ab8] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:26:22,694 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-95) hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:26:22,773 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-95) SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:26:22,776 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-95) starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:26:22,793 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-95) START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 24ba9e54 >2012-11-08 09:26:22,818 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-95) spmStart polling started: taskId = 2845298d-4ce6-4d18-ae52-964c6ca3d15e >2012-11-08 09:26:23,850 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-95) Failed in HSMGetTaskStatusVDS method >2012-11-08 09:26:23,850 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-95) Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:26:23,850 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-95) spmStart polling ended: taskId = 2845298d-4ce6-4d18-ae52-964c6ca3d15e task status = finished >2012-11-08 09:26:23,850 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-95) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:26:23,874 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-95) spmStart polling ended, spm status: Free >2012-11-08 09:26:23,888 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-95) START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=2845298d-4ce6-4d18-ae52-964c6ca3d15e), log id: 405d671e >2012-11-08 09:26:23,905 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-95) FINISH, HSMClearTaskVDSCommand, log id: 405d671e >2012-11-08 09:26:23,905 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-95) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@341d0395, log id: 24ba9e54 >2012-11-08 09:26:23,908 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-95) [41e3afd7] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:26:23,938 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-95) [41e3afd7] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:26:26,713 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-106) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='(~vdsm~, ~vdsm-cli~)' message='vdsm-4.9.6-41.0.el6_3.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:26:29,117 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-106) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='libjpeg' message='libjpeg-6b-46.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:26:31,570 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-106) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='spice-server' message='spice-server-0.10.1-10.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:26:33,967 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-106) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='pixman' message='pixman-0.18.4-1.el6_0.1.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:26:33,998 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:26:34,024 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:26:34,028 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:26:34,044 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-39) START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 670fa30c >2012-11-08 09:26:34,067 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-39) spmStart polling started: taskId = 91312358-c6ca-4816-8dc0-e426dd951ed7 >2012-11-08 09:26:35,102 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-39) Failed in HSMGetTaskStatusVDS method >2012-11-08 09:26:35,102 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-39) Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:26:35,102 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-39) spmStart polling ended: taskId = 91312358-c6ca-4816-8dc0-e426dd951ed7 task status = finished >2012-11-08 09:26:35,102 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-39) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:26:35,133 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-39) spmStart polling ended, spm status: Free >2012-11-08 09:26:35,151 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-39) START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=91312358-c6ca-4816-8dc0-e426dd951ed7), log id: 42d2cb17 >2012-11-08 09:26:35,172 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-39) FINISH, HSMClearTaskVDSCommand, log id: 42d2cb17 >2012-11-08 09:26:35,172 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-39) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1d137547, log id: 670fa30c >2012-11-08 09:26:35,174 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-39) [6878973a] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:26:35,197 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [6878973a] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:26:36,357 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-106) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='seabios' message='seabios-0.6.1.2-19.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:26:38,804 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-106) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='fence-agents' message='fence-agents-3.1.5-17.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:26:41,293 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-106) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='libselinux-python' message='libselinux-python-2.0.94-5.3.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:26:41,843 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-110) Installation of 10.70.36.19. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='(~vdsm~, ~vdsm-cli~)' message='vdsm-4.9.6-41.0.el6_3.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:26:43,694 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-106) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='sanlock' message='sanlock-2.3-4.el6_3.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:26:44,232 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-110) Installation of 10.70.36.19. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='libjpeg' message='libjpeg-6b-46.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:26:45,254 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-56) hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:26:45,282 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-56) SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:26:45,284 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-56) starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:26:45,302 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-56) START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2f8b2135 >2012-11-08 09:26:45,331 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-56) spmStart polling started: taskId = c4365584-ad63-4bb8-8ab3-e9c2370b5611 >2012-11-08 09:26:46,145 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-106) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='sanlock-python' message='sanlock-python-2.3-4.el6_3.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:26:46,371 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-56) Failed in HSMGetTaskStatusVDS method >2012-11-08 09:26:46,371 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-56) Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:26:46,371 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-56) spmStart polling ended: taskId = c4365584-ad63-4bb8-8ab3-e9c2370b5611 task status = finished >2012-11-08 09:26:46,371 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-56) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:26:46,397 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-56) spmStart polling ended, spm status: Free >2012-11-08 09:26:46,413 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-56) START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=c4365584-ad63-4bb8-8ab3-e9c2370b5611), log id: 1f05cb5c >2012-11-08 09:26:46,438 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-56) FINISH, HSMClearTaskVDSCommand, log id: 1f05cb5c >2012-11-08 09:26:46,438 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-56) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@356a14ed, log id: 2f8b2135 >2012-11-08 09:26:46,440 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-56) [4783ef00] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:26:46,458 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-56) [4783ef00] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:26:46,674 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-110) Installation of 10.70.36.19. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='spice-server' message='spice-server-0.10.1-10.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:26:48,597 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-106) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='qemu-img' message='package qemu-img is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:26:49,073 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-110) Installation of 10.70.36.19. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='pixman' message='pixman-0.18.4-1.el6_0.1.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:26:51,045 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-106) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='qemu-kvm-tools' message='package qemu-kvm-tools is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:26:51,474 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-110) Installation of 10.70.36.19. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='seabios' message='seabios-0.6.1.2-19.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:26:53,498 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-106) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='qemu-kvm' message='package qemu-kvm is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:26:53,504 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-106) Installation of 10.70.36.16. Received message: <BSTRAP component='CreateConf' status='OK' message='Basic configuration found, skipping this step'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:26:53,513 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-106) Installation of 10.70.36.16. Received message: <BSTRAP component='CheckLocalHostname' status='OK' message='Local hostname is correct.'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:26:53,564 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-106) Installation of 10.70.36.16. Received message: <BSTRAP component='SetNetworking' status='OK' message='Bridge management already exists. Skipping bridge creation.'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:26:53,679 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-106) Installation of 10.70.36.16. Received message: <BSTRAP component='SetSSHAccess' status='OK' message='SUCCESS'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:26:53,865 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-106) Installation of 10.70.36.16. Received message: <BSTRAP component='VerifyServices' status='OK' message='Needed services set'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:26:53,918 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-110) Installation of 10.70.36.19. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='fence-agents' message='fence-agents-3.1.5-17.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:26:54,582 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-106) Installation of 10.70.36.16. Received message: <BSTRAP component='Encryption setup' status='OK' message='Ended successfully'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:26:54,587 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-106) Installation of 10.70.36.16. Received message: <BSTRAP component='RHEV_INSTALL' status='OK'/>. Stage completed. (Stage: Running first installation script on Host) >2012-11-08 09:26:54,616 ERROR [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-47) [488d048b] Installation of 10.70.36.16. Error: Repository JBOSS is listed more than once in the configuration >Repository qa-latest-5 is listed more than once in the configuration >Repository rhel-source is listed more than once in the configuration >Repository rhel-source-beta is listed more than once in the configuration >Repository rhs-client is listed more than once in the configuration >Repository JBOSS is listed more than once in the configuration >Repository qa-latest-5 is listed more than once in the configuration >Repository rhel-source is listed more than once in the configuration >Repository rhel-source-beta is listed more than once in the configuration >Repository rhs-client is listed more than once in the configuration >. (Stage: Downloading certificate request from Host) >2012-11-08 09:26:54,616 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-47) [488d048b] VDS message: Repository JBOSS is listed more than once in the configuration >2012-11-08 09:26:54,616 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-47) [488d048b] VDS message: Repository qa-latest-5 is listed more than once in the configuration >2012-11-08 09:26:54,616 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-47) [488d048b] VDS message: Repository rhel-source is listed more than once in the configuration >2012-11-08 09:26:54,616 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-47) [488d048b] VDS message: Repository rhel-source-beta is listed more than once in the configuration >2012-11-08 09:26:54,616 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-47) [488d048b] VDS message: Repository rhs-client is listed more than once in the configuration >2012-11-08 09:26:54,616 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-47) [488d048b] VDS message: Repository JBOSS is listed more than once in the configuration >2012-11-08 09:26:54,616 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-47) [488d048b] VDS message: Repository qa-latest-5 is listed more than once in the configuration >2012-11-08 09:26:54,616 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-47) [488d048b] VDS message: Repository rhel-source is listed more than once in the configuration >2012-11-08 09:26:54,616 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-47) [488d048b] VDS message: Repository rhel-source-beta is listed more than once in the configuration >2012-11-08 09:26:54,616 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-47) [488d048b] VDS message: Repository rhs-client is listed more than once in the configuration >2012-11-08 09:26:54,616 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-47) [488d048b] Script ended, result is {1} >2012-11-08 09:26:54,616 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-47) [488d048b] Installation of 10.70.36.16. Executing installation stage. (Stage: Downloading certificate request from Host) >2012-11-08 09:26:54,616 INFO [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-4-thread-47) SSH receive 10.70.36.16:'/tmp/cert_4f881ef1-f92f-411f-881d-f008966c8670.req' '/etc/pki/ovirt-engine/requests/cert_4f881ef1-f92f-411f-881d-f008966c8670.req') >2012-11-08 09:26:54,692 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-47) [488d048b] Installation of 10.70.36.16. successfully done sftp operation ( Stage: Downloading certificate request from Host) >2012-11-08 09:26:54,692 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-47) [488d048b] DownloadCertificateRequest ended:true >2012-11-08 09:26:54,692 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-47) [488d048b] Installation of 10.70.36.16. Executing installation stage. (Stage: Sign certificate request and generate certificate) >2012-11-08 09:26:55,702 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-47) [488d048b] SignCertificateRequest ended:true >2012-11-08 09:26:55,702 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-47) [488d048b] Installation of 10.70.36.16. Executing installation stage. (Stage: Upload signed sertificate to Host) >2012-11-08 09:26:55,723 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-47) [488d048b] Installation of 10.70.36.16. successfully done sftp operation ( Stage: Upload signed sertificate to Host) >2012-11-08 09:26:55,723 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-47) [488d048b] UploadSignedCertificate ended:true >2012-11-08 09:26:55,723 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-47) [488d048b] Installation of 10.70.36.16. Executing installation stage. (Stage: Upload Cerficate Autority to Host) >2012-11-08 09:26:55,777 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-47) [488d048b] Installation of 10.70.36.16. successfully done sftp operation ( Stage: Upload Cerficate Autority to Host) >2012-11-08 09:26:55,777 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-47) [488d048b] Installation of 10.70.36.16. Executing installation stage. (Stage: Running second installation script on Host) >2012-11-08 09:26:55,777 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-47) [488d048b] Installation of 10.70.36.16. Sending SSH Command umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar -C "${MYTMP}" --no-same-permissions -o -x && "${MYTMP}"/setup -c 'ssl=true;management_port=54321' -O 'lab.eng.blr.redhat.com' -t 2012-11-08T14:25:27 -S /tmp/ovirt-id_rsa_4f881ef1-f92f-411f-881d-f008966c8670 -p 80 -b -B rhevm http://guido.lab.eng.blr.redhat.com:80/Components/vds/ http://guido.lab.eng.blr.redhat.com:80/Components/vds/ 10.70.36.16 4f881ef1-f92f-411f-881d-f008966c8670 True < /var/cache/ovirt-engine/vdsm-bootstrap-2.tar. (Stage: Running second installation script on Host) >2012-11-08 09:26:55,778 INFO [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-4-thread-47) SSH execute 10.70.36.16 'umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar -C "${MYTMP}" --no-same-permissions -o -x && "${MYTMP}"/setup -c 'ssl=true;management_port=54321' -O 'lab.eng.blr.redhat.com' -t 2012-11-08T14:25:27 -S /tmp/ovirt-id_rsa_4f881ef1-f92f-411f-881d-f008966c8670 -p 80 -b -B rhevm http://guido.lab.eng.blr.redhat.com:80/Components/vds/ http://guido.lab.eng.blr.redhat.com:80/Components/vds/ 10.70.36.16 4f881ef1-f92f-411f-881d-f008966c8670 True' >2012-11-08 09:26:55,866 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-106) Installation of 10.70.36.16. Received message: <BSTRAP component='INSTALLER' status='OK' message='Test platform succeeded'/>. FYI. (Stage: Running second installation script on Host) >2012-11-08 09:26:56,174 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-106) Installation of 10.70.36.16. Received message: <BSTRAP component='instCert' status='OK'/>. FYI. (Stage: Running second installation script on Host) >2012-11-08 09:26:56,178 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-106) Installation of 10.70.36.16. Received message: <BSTRAP component='CoreDump' status='OK'/>. FYI. (Stage: Running second installation script on Host) >2012-11-08 09:26:56,183 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-106) Installation of 10.70.36.16. Received message: <BSTRAP component='cleanAll' status='OK'/>. FYI. (Stage: Running second installation script on Host) >2012-11-08 09:26:56,188 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-106) Installation of 10.70.36.16. Received message: <BSTRAP component='VDS Configuration' status='OK'/>. FYI. (Stage: Running second installation script on Host) >2012-11-08 09:26:56,193 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-106) Installation of 10.70.36.16. Received message: <BSTRAP component='RHEV_INSTALL' status='OK'/>. Stage completed. (Stage: Running second installation script on Host) >2012-11-08 09:26:56,202 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-106) Installation of 10.70.36.16. Received message: <BSTRAP component='Reboot' status='OK' message='Rebooting machine' />. FYI. (Stage: Host installation complete) >2012-11-08 09:26:56,220 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-47) [488d048b] Script ended, result is {1} >2012-11-08 09:26:56,220 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-4-thread-47) [488d048b] After Installation pool-4-thread-47 >2012-11-08 09:26:56,221 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-47) [488d048b] START, SetVdsStatusVDSCommand(HostName = rhs-gp-srv12.lab.eng.blr.redhat.com, HostId = 056a9890-29af-11e2-b5a8-52540035ed48, status=Reboot, nonOperationalReason=NONE), log id: 6aa37dde >2012-11-08 09:26:56,225 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-47) [488d048b] FINISH, SetVdsStatusVDSCommand, log id: 6aa37dde >2012-11-08 09:26:56,225 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-4-thread-50) [488d048b] Waiting 300 seconds, for server to finish reboot process. >2012-11-08 09:26:56,367 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-110) Installation of 10.70.36.19. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='libselinux-python' message='libselinux-python-2.0.94-5.3.el6.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:26:56,524 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:26:56,552 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:26:56,554 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:26:56,572 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4bf0fadf >2012-11-08 09:26:56,601 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) spmStart polling started: taskId = 41b9c648-59d4-45e5-8634-70b68079df96 >2012-11-08 09:26:57,634 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-5) Failed in HSMGetTaskStatusVDS method >2012-11-08 09:26:57,634 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-5) Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:26:57,634 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) spmStart polling ended: taskId = 41b9c648-59d4-45e5-8634-70b68079df96 task status = finished >2012-11-08 09:26:57,634 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:26:57,662 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) spmStart polling ended, spm status: Free >2012-11-08 09:26:57,676 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-5) START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=41b9c648-59d4-45e5-8634-70b68079df96), log id: 71fb69df >2012-11-08 09:26:57,699 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-5) FINISH, HSMClearTaskVDSCommand, log id: 71fb69df >2012-11-08 09:26:57,699 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@30e3e9ef, log id: 4bf0fadf >2012-11-08 09:26:57,701 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-5) [431362fc] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:26:57,726 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [431362fc] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:26:58,768 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-110) Installation of 10.70.36.19. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='sanlock' message='sanlock-2.3-4.el6_3.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:27:01,217 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-110) Installation of 10.70.36.19. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='sanlock-python' message='sanlock-python-2.3-4.el6_3.x86_64 '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:27:03,669 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-110) Installation of 10.70.36.19. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='qemu-img' message='package qemu-img is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:27:06,117 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-110) Installation of 10.70.36.19. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='qemu-kvm-tools' message='package qemu-kvm-tools is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:27:07,787 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:27:07,814 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:27:07,817 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:27:07,833 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-21) START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: a55ce26 >2012-11-08 09:27:07,857 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-21) spmStart polling started: taskId = a97d9fbd-84aa-42fb-bbd1-48fe7358bf05 >2012-11-08 09:27:08,572 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-110) Installation of 10.70.36.19. Received message: <BSTRAP component='VDS PACKAGES' status='OK' result='qemu-kvm' message='package qemu-kvm is not installed '/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:27:08,577 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-110) Installation of 10.70.36.19. Received message: <BSTRAP component='CreateConf' status='OK' message='Basic configuration found, skipping this step'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:27:08,581 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-110) Installation of 10.70.36.19. Received message: <BSTRAP component='CheckLocalHostname' status='OK' message='Local hostname is correct.'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:27:08,635 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-110) Installation of 10.70.36.19. Received message: <BSTRAP component='SetNetworking' status='OK' message='Bridge management already exists. Skipping bridge creation.'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:27:08,748 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-110) Installation of 10.70.36.19. Received message: <BSTRAP component='SetSSHAccess' status='OK' message='SUCCESS'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:27:08,893 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-21) Failed in HSMGetTaskStatusVDS method >2012-11-08 09:27:08,893 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-21) Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:27:08,893 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-21) spmStart polling ended: taskId = a97d9fbd-84aa-42fb-bbd1-48fe7358bf05 task status = finished >2012-11-08 09:27:08,893 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-21) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:27:08,923 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-21) spmStart polling ended, spm status: Free >2012-11-08 09:27:08,933 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-110) Installation of 10.70.36.19. Received message: <BSTRAP component='VerifyServices' status='OK' message='Needed services set'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:27:08,938 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-21) START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=a97d9fbd-84aa-42fb-bbd1-48fe7358bf05), log id: 1bf5c294 >2012-11-08 09:27:08,960 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-21) FINISH, HSMClearTaskVDSCommand, log id: 1bf5c294 >2012-11-08 09:27:08,960 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-21) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@23b990e3, log id: a55ce26 >2012-11-08 09:27:08,962 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-21) [6bd73967] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:27:08,986 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) [6bd73967] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:27:19,053 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:27:19,082 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:27:19,084 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:27:19,102 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 169e922c >2012-11-08 09:27:19,130 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) spmStart polling started: taskId = 7af03d7e-943b-4f78-a4ba-fd5dacc883e0 >2012-11-08 09:27:20,170 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-66) Failed in HSMGetTaskStatusVDS method >2012-11-08 09:27:20,170 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-66) Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:27:20,170 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) spmStart polling ended: taskId = 7af03d7e-943b-4f78-a4ba-fd5dacc883e0 task status = finished >2012-11-08 09:27:20,170 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:27:20,197 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) spmStart polling ended, spm status: Free >2012-11-08 09:27:20,208 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-66) START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=7af03d7e-943b-4f78-a4ba-fd5dacc883e0), log id: 56f0f4ae >2012-11-08 09:27:20,229 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-66) FINISH, HSMClearTaskVDSCommand, log id: 56f0f4ae >2012-11-08 09:27:20,229 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@654ff819, log id: 169e922c >2012-11-08 09:27:20,231 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-66) [65296545] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:27:20,257 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [65296545] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:27:21,602 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-110) Installation of 10.70.36.19. Received message: <BSTRAP component='Encryption setup' status='OK' message='Ended successfully'/>. FYI. (Stage: Running first installation script on Host) >2012-11-08 09:27:21,607 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-110) Installation of 10.70.36.19. Received message: <BSTRAP component='RHEV_INSTALL' status='OK'/>. Stage completed. (Stage: Running first installation script on Host) >2012-11-08 09:27:21,639 ERROR [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-45) [28e3a2cf] Installation of 10.70.36.19. Error: Repository JBOSS is listed more than once in the configuration >Repository qa-latest-5 is listed more than once in the configuration >Repository rhel-source is listed more than once in the configuration >Repository rhel-source-beta is listed more than once in the configuration >Repository rhs-client is listed more than once in the configuration >Repository JBOSS is listed more than once in the configuration >Repository qa-latest-5 is listed more than once in the configuration >Repository rhel-source is listed more than once in the configuration >Repository rhel-source-beta is listed more than once in the configuration >Repository rhs-client is listed more than once in the configuration >. (Stage: Downloading certificate request from Host) >2012-11-08 09:27:21,639 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-45) [28e3a2cf] VDS message: Repository JBOSS is listed more than once in the configuration >2012-11-08 09:27:21,639 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-45) [28e3a2cf] VDS message: Repository qa-latest-5 is listed more than once in the configuration >2012-11-08 09:27:21,639 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-45) [28e3a2cf] VDS message: Repository rhel-source is listed more than once in the configuration >2012-11-08 09:27:21,639 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-45) [28e3a2cf] VDS message: Repository rhel-source-beta is listed more than once in the configuration >2012-11-08 09:27:21,639 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-45) [28e3a2cf] VDS message: Repository rhs-client is listed more than once in the configuration >2012-11-08 09:27:21,639 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-45) [28e3a2cf] VDS message: Repository JBOSS is listed more than once in the configuration >2012-11-08 09:27:21,640 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-45) [28e3a2cf] VDS message: Repository qa-latest-5 is listed more than once in the configuration >2012-11-08 09:27:21,640 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-45) [28e3a2cf] VDS message: Repository rhel-source is listed more than once in the configuration >2012-11-08 09:27:21,640 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-45) [28e3a2cf] VDS message: Repository rhel-source-beta is listed more than once in the configuration >2012-11-08 09:27:21,640 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-45) [28e3a2cf] VDS message: Repository rhs-client is listed more than once in the configuration >2012-11-08 09:27:21,640 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-45) [28e3a2cf] Script ended, result is {1} >2012-11-08 09:27:21,640 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-45) [28e3a2cf] Installation of 10.70.36.19. Executing installation stage. (Stage: Downloading certificate request from Host) >2012-11-08 09:27:21,640 INFO [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-4-thread-45) SSH receive 10.70.36.19:'/tmp/cert_606a7cd9-9517-4da1-9a6d-c6b19fb18b2e.req' '/etc/pki/ovirt-engine/requests/cert_606a7cd9-9517-4da1-9a6d-c6b19fb18b2e.req') >2012-11-08 09:27:21,709 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-45) [28e3a2cf] Installation of 10.70.36.19. successfully done sftp operation ( Stage: Downloading certificate request from Host) >2012-11-08 09:27:21,709 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-45) [28e3a2cf] DownloadCertificateRequest ended:true >2012-11-08 09:27:21,709 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-45) [28e3a2cf] Installation of 10.70.36.19. Executing installation stage. (Stage: Sign certificate request and generate certificate) >2012-11-08 09:27:22,711 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-45) [28e3a2cf] SignCertificateRequest ended:true >2012-11-08 09:27:22,711 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-45) [28e3a2cf] Installation of 10.70.36.19. Executing installation stage. (Stage: Upload signed sertificate to Host) >2012-11-08 09:27:22,735 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-45) [28e3a2cf] Installation of 10.70.36.19. successfully done sftp operation ( Stage: Upload signed sertificate to Host) >2012-11-08 09:27:22,735 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-45) [28e3a2cf] UploadSignedCertificate ended:true >2012-11-08 09:27:22,735 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-45) [28e3a2cf] Installation of 10.70.36.19. Executing installation stage. (Stage: Upload Cerficate Autority to Host) >2012-11-08 09:27:22,789 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-45) [28e3a2cf] Installation of 10.70.36.19. successfully done sftp operation ( Stage: Upload Cerficate Autority to Host) >2012-11-08 09:27:22,789 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-45) [28e3a2cf] Installation of 10.70.36.19. Executing installation stage. (Stage: Running second installation script on Host) >2012-11-08 09:27:22,789 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-45) [28e3a2cf] Installation of 10.70.36.19. Sending SSH Command umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar -C "${MYTMP}" --no-same-permissions -o -x && "${MYTMP}"/setup -c 'ssl=true;management_port=54321' -O 'lab.eng.blr.redhat.com' -t 2012-11-08T14:25:42 -S /tmp/ovirt-id_rsa_606a7cd9-9517-4da1-9a6d-c6b19fb18b2e -p 80 -b -B rhevm http://guido.lab.eng.blr.redhat.com:80/Components/vds/ http://guido.lab.eng.blr.redhat.com:80/Components/vds/ 10.70.36.19 606a7cd9-9517-4da1-9a6d-c6b19fb18b2e True < /var/cache/ovirt-engine/vdsm-bootstrap-2.tar. (Stage: Running second installation script on Host) >2012-11-08 09:27:22,790 INFO [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-4-thread-45) SSH execute 10.70.36.19 'umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar -C "${MYTMP}" --no-same-permissions -o -x && "${MYTMP}"/setup -c 'ssl=true;management_port=54321' -O 'lab.eng.blr.redhat.com' -t 2012-11-08T14:25:42 -S /tmp/ovirt-id_rsa_606a7cd9-9517-4da1-9a6d-c6b19fb18b2e -p 80 -b -B rhevm http://guido.lab.eng.blr.redhat.com:80/Components/vds/ http://guido.lab.eng.blr.redhat.com:80/Components/vds/ 10.70.36.19 606a7cd9-9517-4da1-9a6d-c6b19fb18b2e True' >2012-11-08 09:27:22,881 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-110) Installation of 10.70.36.19. Received message: <BSTRAP component='INSTALLER' status='OK' message='Test platform succeeded'/>. FYI. (Stage: Running second installation script on Host) >2012-11-08 09:27:23,208 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-110) Installation of 10.70.36.19. Received message: <BSTRAP component='instCert' status='OK'/>. FYI. (Stage: Running second installation script on Host) >2012-11-08 09:27:23,212 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-110) Installation of 10.70.36.19. Received message: <BSTRAP component='CoreDump' status='OK'/>. FYI. (Stage: Running second installation script on Host) >2012-11-08 09:27:23,214 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-110) Installation of 10.70.36.19. Received message: <BSTRAP component='cleanAll' status='OK'/>. FYI. (Stage: Running second installation script on Host) >2012-11-08 09:27:23,217 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-110) Installation of 10.70.36.19. Received message: <BSTRAP component='VDS Configuration' status='OK'/>. FYI. (Stage: Running second installation script on Host) >2012-11-08 09:27:23,220 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-110) Installation of 10.70.36.19. Received message: <BSTRAP component='RHEV_INSTALL' status='OK'/>. Stage completed. (Stage: Running second installation script on Host) >2012-11-08 09:27:23,241 INFO [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-110) Installation of 10.70.36.19. Received message: <BSTRAP component='Reboot' status='OK' message='Rebooting machine' />. FYI. (Stage: Host installation complete) >2012-11-08 09:27:23,257 INFO [org.ovirt.engine.core.bll.VdsInstaller] (pool-4-thread-45) [28e3a2cf] Script ended, result is {1} >2012-11-08 09:27:23,258 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-4-thread-45) [28e3a2cf] After Installation pool-4-thread-45 >2012-11-08 09:27:23,260 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-45) [28e3a2cf] START, SetVdsStatusVDSCommand(HostName = rhs-gp-srv15.lab.eng.blr.redhat.com, HostId = 12e4f376-29af-11e2-b479-52540035ed48, status=Reboot, nonOperationalReason=NONE), log id: 184a99fe >2012-11-08 09:27:23,264 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-45) [28e3a2cf] FINISH, SetVdsStatusVDSCommand, log id: 184a99fe >2012-11-08 09:27:23,264 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-4-thread-47) [28e3a2cf] Waiting 300 seconds, for server to finish reboot process. >2012-11-08 09:27:30,314 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:27:30,341 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:27:30,344 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:27:30,361 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-96) START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 31ff4b1d >2012-11-08 09:27:30,390 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-96) spmStart polling started: taskId = 51755721-baa6-4e9a-920d-7d3566997027 >2012-11-08 09:27:31,427 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-96) Failed in HSMGetTaskStatusVDS method >2012-11-08 09:27:31,427 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-96) Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:27:31,427 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-96) spmStart polling ended: taskId = 51755721-baa6-4e9a-920d-7d3566997027 task status = finished >2012-11-08 09:27:31,427 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-96) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:27:31,454 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-96) spmStart polling ended, spm status: Free >2012-11-08 09:27:31,471 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-96) START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=51755721-baa6-4e9a-920d-7d3566997027), log id: 7fed86d8 >2012-11-08 09:27:31,496 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-96) FINISH, HSMClearTaskVDSCommand, log id: 7fed86d8 >2012-11-08 09:27:31,496 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-96) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@43301085, log id: 31ff4b1d >2012-11-08 09:27:31,498 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-96) [88cae39] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:27:31,517 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [88cae39] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:27:41,578 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-26) hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:27:41,609 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-26) SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:27:41,611 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-26) starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:27:41,630 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-26) START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 38362851 >2012-11-08 09:27:41,655 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-26) spmStart polling started: taskId = e20b90ba-c453-4b82-887a-df932c73c70c >2012-11-08 09:27:42,699 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-26) Failed in HSMGetTaskStatusVDS method >2012-11-08 09:27:42,699 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-26) Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:27:42,699 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-26) spmStart polling ended: taskId = e20b90ba-c453-4b82-887a-df932c73c70c task status = finished >2012-11-08 09:27:42,699 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-26) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:27:42,731 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-26) spmStart polling ended, spm status: Free >2012-11-08 09:27:42,749 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-26) START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=e20b90ba-c453-4b82-887a-df932c73c70c), log id: 3e0c957d >2012-11-08 09:27:42,774 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-26) FINISH, HSMClearTaskVDSCommand, log id: 3e0c957d >2012-11-08 09:27:42,774 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-26) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@64e8d9ef, log id: 38362851 >2012-11-08 09:27:42,776 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-26) [733c0daa] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:27:42,803 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-26) [733c0daa] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:27:52,868 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:27:52,899 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:27:52,902 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:27:52,914 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-57) START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2337c016 >2012-11-08 09:27:52,941 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-57) spmStart polling started: taskId = e4a0ff01-0542-4950-ab49-e5f76c983c93 >2012-11-08 09:27:53,977 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-57) Failed in HSMGetTaskStatusVDS method >2012-11-08 09:27:53,977 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-57) Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:27:53,977 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-57) spmStart polling ended: taskId = e4a0ff01-0542-4950-ab49-e5f76c983c93 task status = finished >2012-11-08 09:27:53,977 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-57) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:27:54,009 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-57) spmStart polling ended, spm status: Free >2012-11-08 09:27:54,028 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-57) START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=e4a0ff01-0542-4950-ab49-e5f76c983c93), log id: 7a6e5218 >2012-11-08 09:27:54,053 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-57) FINISH, HSMClearTaskVDSCommand, log id: 7a6e5218 >2012-11-08 09:27:54,053 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-57) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@60cfe473, log id: 2337c016 >2012-11-08 09:27:54,055 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-57) [709f6bac] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:27:54,082 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [709f6bac] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:28:04,141 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-62) hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:28:04,169 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-62) SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:28:04,172 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-62) starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:28:04,189 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-62) START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 74261e7d >2012-11-08 09:28:04,210 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-62) spmStart polling started: taskId = 88ee8d3e-be4e-4b5a-81de-bb8e3bc76975 >2012-11-08 09:28:05,245 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-62) Failed in HSMGetTaskStatusVDS method >2012-11-08 09:28:05,245 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-62) Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:28:05,245 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-62) spmStart polling ended: taskId = 88ee8d3e-be4e-4b5a-81de-bb8e3bc76975 task status = finished >2012-11-08 09:28:05,245 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-62) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:28:05,273 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-62) spmStart polling ended, spm status: Free >2012-11-08 09:28:05,285 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-62) START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=88ee8d3e-be4e-4b5a-81de-bb8e3bc76975), log id: 6dbb5c41 >2012-11-08 09:28:05,307 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-62) FINISH, HSMClearTaskVDSCommand, log id: 6dbb5c41 >2012-11-08 09:28:05,307 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-62) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@27a5e7af, log id: 74261e7d >2012-11-08 09:28:05,309 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-62) [7d0da67e] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:28:05,333 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-62) [7d0da67e] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:28:15,393 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-34) hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:28:15,415 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-34) SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:28:15,417 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-34) starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:28:15,432 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-34) START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 554d647e >2012-11-08 09:28:15,459 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-34) spmStart polling started: taskId = 193553f4-7cb5-44b7-bd9f-dceb92b75178 >2012-11-08 09:28:16,496 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-34) Failed in HSMGetTaskStatusVDS method >2012-11-08 09:28:16,496 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-34) Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:28:16,496 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-34) spmStart polling ended: taskId = 193553f4-7cb5-44b7-bd9f-dceb92b75178 task status = finished >2012-11-08 09:28:16,497 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-34) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:28:16,528 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-34) spmStart polling ended, spm status: Free >2012-11-08 09:28:16,542 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-34) START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=193553f4-7cb5-44b7-bd9f-dceb92b75178), log id: 48ca29a5 >2012-11-08 09:28:16,565 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-34) FINISH, HSMClearTaskVDSCommand, log id: 48ca29a5 >2012-11-08 09:28:16,565 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-34) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@507434f8, log id: 554d647e >2012-11-08 09:28:16,567 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-34) [7d41af0b] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:28:16,584 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-34) [7d41af0b] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:28:26,638 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:28:26,667 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:28:26,670 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:28:26,687 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 507d7fe5 >2012-11-08 09:28:26,717 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) spmStart polling started: taskId = 16f3487b-e1c0-4c45-a816-e90a2e7211b9 >2012-11-08 09:28:27,750 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-61) Failed in HSMGetTaskStatusVDS method >2012-11-08 09:28:27,750 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-61) Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:28:27,750 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) spmStart polling ended: taskId = 16f3487b-e1c0-4c45-a816-e90a2e7211b9 task status = finished >2012-11-08 09:28:27,750 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:28:27,777 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) spmStart polling ended, spm status: Free >2012-11-08 09:28:27,796 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-61) START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=16f3487b-e1c0-4c45-a816-e90a2e7211b9), log id: 26116de9 >2012-11-08 09:28:27,812 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-61) FINISH, HSMClearTaskVDSCommand, log id: 26116de9 >2012-11-08 09:28:27,812 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4246ef85, log id: 507d7fe5 >2012-11-08 09:28:27,815 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-61) [3f1726c1] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:28:27,836 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [3f1726c1] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:28:37,890 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:28:37,916 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:28:37,919 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:28:37,932 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-84) START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 31cd2d6a >2012-11-08 09:28:37,955 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-84) spmStart polling started: taskId = 784d2c13-5a75-4bdb-9289-7b2171ae158a >2012-11-08 09:28:38,988 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-84) Failed in HSMGetTaskStatusVDS method >2012-11-08 09:28:38,988 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-84) Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:28:38,988 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-84) spmStart polling ended: taskId = 784d2c13-5a75-4bdb-9289-7b2171ae158a task status = finished >2012-11-08 09:28:38,988 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-84) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:28:39,017 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-84) spmStart polling ended, spm status: Free >2012-11-08 09:28:39,031 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-84) START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=784d2c13-5a75-4bdb-9289-7b2171ae158a), log id: 8eaeed >2012-11-08 09:28:39,049 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-84) FINISH, HSMClearTaskVDSCommand, log id: 8eaeed >2012-11-08 09:28:39,049 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-84) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@177e49e4, log id: 31cd2d6a >2012-11-08 09:28:39,051 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-84) [642523c8] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:28:39,070 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [642523c8] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:28:49,130 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-22) hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:28:49,158 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-22) SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:28:49,161 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-22) starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:28:49,178 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-22) START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 10d15fe1 >2012-11-08 09:28:49,207 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-22) spmStart polling started: taskId = f0e80909-5cc4-40c6-a8c8-a70419c92efe >2012-11-08 09:28:50,244 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-22) Failed in HSMGetTaskStatusVDS method >2012-11-08 09:28:50,244 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-22) Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:28:50,244 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-22) spmStart polling ended: taskId = f0e80909-5cc4-40c6-a8c8-a70419c92efe task status = finished >2012-11-08 09:28:50,244 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-22) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:28:50,276 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-22) spmStart polling ended, spm status: Free >2012-11-08 09:28:50,290 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-22) START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=f0e80909-5cc4-40c6-a8c8-a70419c92efe), log id: 5845589f >2012-11-08 09:28:50,313 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-22) FINISH, HSMClearTaskVDSCommand, log id: 5845589f >2012-11-08 09:28:50,313 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-22) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@30bd5bff, log id: 10d15fe1 >2012-11-08 09:28:50,315 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-22) [5fe2db58] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:28:50,332 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-22) [5fe2db58] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:29:00,387 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-68) hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:29:00,416 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-68) SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:29:00,419 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-68) starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:29:00,437 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-68) START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 288d7199 >2012-11-08 09:29:00,460 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-68) spmStart polling started: taskId = f3f5aaa4-fc78-4dc2-a269-a87c6568e075 >2012-11-08 09:29:01,494 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-68) Failed in HSMGetTaskStatusVDS method >2012-11-08 09:29:01,494 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-68) Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:29:01,494 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-68) spmStart polling ended: taskId = f3f5aaa4-fc78-4dc2-a269-a87c6568e075 task status = finished >2012-11-08 09:29:01,494 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-68) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:29:01,526 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-68) spmStart polling ended, spm status: Free >2012-11-08 09:29:01,545 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-68) START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=f3f5aaa4-fc78-4dc2-a269-a87c6568e075), log id: 6a922b2e >2012-11-08 09:29:01,562 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-68) FINISH, HSMClearTaskVDSCommand, log id: 6a922b2e >2012-11-08 09:29:01,562 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-68) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@22510fab, log id: 288d7199 >2012-11-08 09:29:01,564 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-68) [77dbb526] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:29:01,590 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-68) [77dbb526] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:29:11,656 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-91) hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:29:11,680 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-91) SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:29:11,683 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-91) starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:29:11,698 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-91) START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 44d686f2 >2012-11-08 09:29:11,726 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-91) spmStart polling started: taskId = fb18709a-3015-431c-8ea2-ae6365e851a4 >2012-11-08 09:29:12,771 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-91) Failed in HSMGetTaskStatusVDS method >2012-11-08 09:29:12,771 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-91) Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:29:12,771 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-91) spmStart polling ended: taskId = fb18709a-3015-431c-8ea2-ae6365e851a4 task status = finished >2012-11-08 09:29:12,771 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-91) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:29:12,799 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-91) spmStart polling ended, spm status: Free >2012-11-08 09:29:12,811 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-91) START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=fb18709a-3015-431c-8ea2-ae6365e851a4), log id: 551554aa >2012-11-08 09:29:12,833 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-91) FINISH, HSMClearTaskVDSCommand, log id: 551554aa >2012-11-08 09:29:12,833 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-91) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6b699089, log id: 44d686f2 >2012-11-08 09:29:12,835 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-91) [5f904976] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:29:12,853 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-91) [5f904976] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:29:22,923 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:29:22,953 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:29:22,956 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:29:22,975 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: ba4247a >2012-11-08 09:29:23,000 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) spmStart polling started: taskId = 3fb5476c-bb4b-42e7-b90b-6232930e622d >2012-11-08 09:29:24,036 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-25) Failed in HSMGetTaskStatusVDS method >2012-11-08 09:29:24,036 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-25) Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:29:24,036 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) spmStart polling ended: taskId = 3fb5476c-bb4b-42e7-b90b-6232930e622d task status = finished >2012-11-08 09:29:24,036 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:29:24,063 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) spmStart polling ended, spm status: Free >2012-11-08 09:29:24,082 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-25) START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=3fb5476c-bb4b-42e7-b90b-6232930e622d), log id: 7e0cbe9b >2012-11-08 09:29:24,108 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-25) FINISH, HSMClearTaskVDSCommand, log id: 7e0cbe9b >2012-11-08 09:29:24,108 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@76b5f2c3, log id: ba4247a >2012-11-08 09:29:24,110 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-25) [23551d18] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:29:24,131 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [23551d18] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:29:34,186 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:29:34,212 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:29:34,215 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:29:34,234 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-50) START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: b1ffb8b >2012-11-08 09:29:34,255 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-50) spmStart polling started: taskId = e4114b68-9751-42e6-914f-31e8458d12c6 >2012-11-08 09:29:35,292 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-50) Failed in HSMGetTaskStatusVDS method >2012-11-08 09:29:35,292 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-50) Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:29:35,292 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-50) spmStart polling ended: taskId = e4114b68-9751-42e6-914f-31e8458d12c6 task status = finished >2012-11-08 09:29:35,292 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-50) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:29:35,318 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-50) spmStart polling ended, spm status: Free >2012-11-08 09:29:35,330 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-50) START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=e4114b68-9751-42e6-914f-31e8458d12c6), log id: 3034ceaf >2012-11-08 09:29:35,348 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-50) FINISH, HSMClearTaskVDSCommand, log id: 3034ceaf >2012-11-08 09:29:35,348 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-50) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@5aed5be2, log id: b1ffb8b >2012-11-08 09:29:35,350 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-50) [152b8111] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:29:35,393 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [152b8111] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:29:45,452 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:29:45,482 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:29:45,484 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:29:45,494 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-86) START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 257c8832 >2012-11-08 09:29:45,516 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-86) spmStart polling started: taskId = 98502250-554b-4f14-b556-d04f53778e33 >2012-11-08 09:29:46,547 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-86) Failed in HSMGetTaskStatusVDS method >2012-11-08 09:29:46,547 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-86) Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:29:46,547 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-86) spmStart polling ended: taskId = 98502250-554b-4f14-b556-d04f53778e33 task status = finished >2012-11-08 09:29:46,547 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-86) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:29:46,575 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-86) spmStart polling ended, spm status: Free >2012-11-08 09:29:46,588 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-86) START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=98502250-554b-4f14-b556-d04f53778e33), log id: 61ee1f70 >2012-11-08 09:29:46,606 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-86) FINISH, HSMClearTaskVDSCommand, log id: 61ee1f70 >2012-11-08 09:29:46,606 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-86) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@5852244a, log id: 257c8832 >2012-11-08 09:29:46,609 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-86) [78cca392] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:29:46,629 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) [78cca392] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:29:54,292 INFO [org.ovirt.engine.core.bll.storage.FenceVdsManualyCommand] (ajp-/127.0.0.1:8702-4) Lock Acquired to object EngineLock [exclusiveLocks= key: 12e4f376-29af-11e2-b479-52540035ed48 value: VDS_FENCE >, sharedLocks= ] >2012-11-08 09:29:54,293 INFO [org.ovirt.engine.core.bll.storage.FenceVdsManualyCommand] (pool-4-thread-45) [692001e2] Running command: FenceVdsManualyCommand internal: false. Entities affected : ID: 12e4f376-29af-11e2-b479-52540035ed48 Type: VDS >2012-11-08 09:29:54,297 INFO [org.ovirt.engine.core.bll.ClearNonResponsiveVdsVmsCommand] (pool-4-thread-45) [263a6ec4] Running command: ClearNonResponsiveVdsVmsCommand internal: true. Entities affected : ID: 12e4f376-29af-11e2-b479-52540035ed48 Type: VDS >2012-11-08 09:29:54,299 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsVMsClearedVDSCommand] (pool-4-thread-45) [263a6ec4] START, UpdateVdsVMsClearedVDSCommand(HostName = rhs-gp-srv15.lab.eng.blr.redhat.com, HostId = 12e4f376-29af-11e2-b479-52540035ed48), log id: 8b68ae9 >2012-11-08 09:29:54,300 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsVMsClearedVDSCommand] (pool-4-thread-45) [263a6ec4] FINISH, UpdateVdsVMsClearedVDSCommand, log id: 8b68ae9 >2012-11-08 09:29:54,307 INFO [org.ovirt.engine.core.bll.storage.FenceVdsManualyCommand] (pool-4-thread-45) [263a6ec4] Lock freed to object EngineLock [exclusiveLocks= key: 12e4f376-29af-11e2-b479-52540035ed48 value: VDS_FENCE >, sharedLocks= ] >2012-11-08 09:29:56,673 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:29:56,702 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:29:56,705 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:29:56,723 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-27) START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: c376fa1 >2012-11-08 09:29:56,754 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-27) spmStart polling started: taskId = 7bcadd28-1449-44d1-a99c-85e011bfd896 >2012-11-08 09:29:57,791 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-27) Failed in HSMGetTaskStatusVDS method >2012-11-08 09:29:57,791 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-27) Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:29:57,791 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-27) spmStart polling ended: taskId = 7bcadd28-1449-44d1-a99c-85e011bfd896 task status = finished >2012-11-08 09:29:57,791 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-27) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:29:57,823 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-27) spmStart polling ended, spm status: Free >2012-11-08 09:29:57,842 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-27) START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=7bcadd28-1449-44d1-a99c-85e011bfd896), log id: acddc2f >2012-11-08 09:29:57,863 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-27) FINISH, HSMClearTaskVDSCommand, log id: acddc2f >2012-11-08 09:29:57,863 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-27) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@51329733, log id: c376fa1 >2012-11-08 09:29:57,865 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-27) [5064beea] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:29:57,881 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [5064beea] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:30:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-39) [6878973a] Autorecovering hosts is disabled, skipping >2012-11-08 09:30:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-39) [6878973a] Autorecovering storage domains is disabled, skipping >2012-11-08 09:30:07,935 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [152b8111] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:30:07,963 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [152b8111] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:30:07,966 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [152b8111] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:30:07,983 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-50) [152b8111] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 380b5bd2 >2012-11-08 09:30:08,005 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-50) [152b8111] spmStart polling started: taskId = ccabf7b7-239e-4316-99f1-6d3caaceba3a >2012-11-08 09:30:09,041 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-50) [152b8111] Failed in HSMGetTaskStatusVDS method >2012-11-08 09:30:09,041 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-50) [152b8111] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:30:09,041 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-50) [152b8111] spmStart polling ended: taskId = ccabf7b7-239e-4316-99f1-6d3caaceba3a task status = finished >2012-11-08 09:30:09,041 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-50) [152b8111] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:30:09,073 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-50) [152b8111] spmStart polling ended, spm status: Free >2012-11-08 09:30:09,087 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-50) [152b8111] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=ccabf7b7-239e-4316-99f1-6d3caaceba3a), log id: 72a9aced >2012-11-08 09:30:09,105 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-50) [152b8111] FINISH, HSMClearTaskVDSCommand, log id: 72a9aced >2012-11-08 09:30:09,105 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-50) [152b8111] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6198366a, log id: 380b5bd2 >2012-11-08 09:30:09,107 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-50) [501c9b34] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:30:09,128 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [501c9b34] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:30:13,930 INFO [org.ovirt.engine.core.bll.storage.FenceVdsManualyCommand] (ajp-/127.0.0.1:8702-2) Lock Acquired to object EngineLock [exclusiveLocks= key: 056a9890-29af-11e2-b5a8-52540035ed48 value: VDS_FENCE >, sharedLocks= ] >2012-11-08 09:30:13,931 INFO [org.ovirt.engine.core.bll.storage.FenceVdsManualyCommand] (pool-4-thread-49) [65c01ff4] Running command: FenceVdsManualyCommand internal: false. Entities affected : ID: 056a9890-29af-11e2-b5a8-52540035ed48 Type: VDS >2012-11-08 09:30:13,936 INFO [org.ovirt.engine.core.bll.ClearNonResponsiveVdsVmsCommand] (pool-4-thread-49) [684b666f] Running command: ClearNonResponsiveVdsVmsCommand internal: true. Entities affected : ID: 056a9890-29af-11e2-b5a8-52540035ed48 Type: VDS >2012-11-08 09:30:13,938 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsVMsClearedVDSCommand] (pool-4-thread-49) [684b666f] START, UpdateVdsVMsClearedVDSCommand(HostName = rhs-gp-srv12.lab.eng.blr.redhat.com, HostId = 056a9890-29af-11e2-b5a8-52540035ed48), log id: 596d196d >2012-11-08 09:30:13,939 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsVMsClearedVDSCommand] (pool-4-thread-49) [684b666f] FINISH, UpdateVdsVMsClearedVDSCommand, log id: 596d196d >2012-11-08 09:30:13,960 INFO [org.ovirt.engine.core.bll.storage.FenceVdsManualyCommand] (pool-4-thread-49) [684b666f] Lock freed to object EngineLock [exclusiveLocks= key: 056a9890-29af-11e2-b5a8-52540035ed48 value: VDS_FENCE >, sharedLocks= ] >2012-11-08 09:30:19,189 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-81) hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:30:19,216 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-81) SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:30:19,219 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-81) starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:30:19,235 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-81) START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3fa28664 >2012-11-08 09:30:19,264 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-81) spmStart polling started: taskId = faac99f9-af42-4c38-97c8-a98dc000769b >2012-11-08 09:30:20,301 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-81) Failed in HSMGetTaskStatusVDS method >2012-11-08 09:30:20,301 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-81) Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:30:20,301 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-81) spmStart polling ended: taskId = faac99f9-af42-4c38-97c8-a98dc000769b task status = finished >2012-11-08 09:30:20,301 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-81) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:30:20,333 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-81) spmStart polling ended, spm status: Free >2012-11-08 09:30:20,346 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-81) START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=faac99f9-af42-4c38-97c8-a98dc000769b), log id: 1be58ea0 >2012-11-08 09:30:20,365 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-81) FINISH, HSMClearTaskVDSCommand, log id: 1be58ea0 >2012-11-08 09:30:20,365 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-81) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7bf6b8ae, log id: 3fa28664 >2012-11-08 09:30:20,366 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-81) [2d6e717] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:30:20,389 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-81) [2d6e717] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:30:30,446 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:30:30,469 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:30:30,471 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:30:30,490 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-13) START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 309fd75d >2012-11-08 09:30:30,521 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-13) spmStart polling started: taskId = a226e72a-71dc-490f-bfbd-38237e4dfa9e >2012-11-08 09:30:31,558 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-13) Failed in HSMGetTaskStatusVDS method >2012-11-08 09:30:31,559 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-13) Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:30:31,559 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-13) spmStart polling ended: taskId = a226e72a-71dc-490f-bfbd-38237e4dfa9e task status = finished >2012-11-08 09:30:31,559 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-13) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:30:31,586 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-13) spmStart polling ended, spm status: Free >2012-11-08 09:30:31,598 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-13) START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=a226e72a-71dc-490f-bfbd-38237e4dfa9e), log id: 333c920e >2012-11-08 09:30:31,624 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-13) FINISH, HSMClearTaskVDSCommand, log id: 333c920e >2012-11-08 09:30:31,624 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-13) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7489c862, log id: 309fd75d >2012-11-08 09:30:31,627 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-13) [1c2d5b4] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:30:31,653 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) [1c2d5b4] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:30:41,713 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:30:41,742 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:30:41,744 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:30:41,759 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-55) START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: e585966 >2012-11-08 09:30:41,784 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-55) spmStart polling started: taskId = b499724e-0dc7-4f89-9af2-9b0f46b7cfc2 >2012-11-08 09:30:42,820 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-55) Failed in HSMGetTaskStatusVDS method >2012-11-08 09:30:42,820 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-55) Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:30:42,820 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-55) spmStart polling ended: taskId = b499724e-0dc7-4f89-9af2-9b0f46b7cfc2 task status = finished >2012-11-08 09:30:42,820 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-55) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:30:42,852 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-55) spmStart polling ended, spm status: Free >2012-11-08 09:30:42,865 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-55) START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=b499724e-0dc7-4f89-9af2-9b0f46b7cfc2), log id: 6c063c2c >2012-11-08 09:30:42,884 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-55) FINISH, HSMClearTaskVDSCommand, log id: 6c063c2c >2012-11-08 09:30:42,884 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-55) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@d58a41c, log id: e585966 >2012-11-08 09:30:42,885 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-55) [367e311c] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:30:42,910 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) [367e311c] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:30:48,782 INFO [org.ovirt.engine.core.bll.SetVmTicketCommand] (ajp-/127.0.0.1:8702-6) [5cfdb101] Running command: SetVmTicketCommand internal: false. Entities affected : ID: 67bd2d56-8e0b-4328-bcfd-ea7f4758c885 Type: VM >2012-11-08 09:30:48,811 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp-/127.0.0.1:8702-6) [5cfdb101] START, SetVmTicketVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=67bd2d56-8e0b-4328-bcfd-ea7f4758c885, ticket=juzJK8IX6CKs, validTime=120,m userName=admin@internal, userId=fdfc627c-d875-11e0-90f0-83df133b58cc), log id: 70934d4f >2012-11-08 09:30:48,849 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp-/127.0.0.1:8702-6) [5cfdb101] FINISH, SetVmTicketVDSCommand, log id: 70934d4f >2012-11-08 09:30:52,963 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:30:52,992 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:30:52,994 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:30:53,007 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-87) START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 937033e >2012-11-08 09:30:53,030 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-87) spmStart polling started: taskId = 09762e4c-bd76-4ae4-afdf-c9f912cfdbb8 >2012-11-08 09:30:54,062 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-87) Failed in HSMGetTaskStatusVDS method >2012-11-08 09:30:54,062 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-87) Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:30:54,062 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-87) spmStart polling ended: taskId = 09762e4c-bd76-4ae4-afdf-c9f912cfdbb8 task status = finished >2012-11-08 09:30:54,062 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-87) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:30:54,091 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-87) spmStart polling ended, spm status: Free >2012-11-08 09:30:54,110 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-87) START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=09762e4c-bd76-4ae4-afdf-c9f912cfdbb8), log id: 70d7ee2e >2012-11-08 09:30:54,128 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-87) FINISH, HSMClearTaskVDSCommand, log id: 70d7ee2e >2012-11-08 09:30:54,128 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-87) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@705647e2, log id: 937033e >2012-11-08 09:30:54,130 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-87) [3089800c] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:30:54,155 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [3089800c] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:31:04,211 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:31:04,236 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:31:04,239 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:31:04,253 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 11000fd9 >2012-11-08 09:31:04,275 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) spmStart polling started: taskId = 4d6d83c2-399b-4ef0-8a7f-9f6a44d82454 >2012-11-08 09:31:05,309 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-23) Failed in HSMGetTaskStatusVDS method >2012-11-08 09:31:05,309 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-23) Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:31:05,309 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) spmStart polling ended: taskId = 4d6d83c2-399b-4ef0-8a7f-9f6a44d82454 task status = finished >2012-11-08 09:31:05,309 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:31:05,340 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) spmStart polling ended, spm status: Free >2012-11-08 09:31:05,359 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-23) START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=4d6d83c2-399b-4ef0-8a7f-9f6a44d82454), log id: 8c9af16 >2012-11-08 09:31:05,376 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-23) FINISH, HSMClearTaskVDSCommand, log id: 8c9af16 >2012-11-08 09:31:05,376 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7105d1db, log id: 11000fd9 >2012-11-08 09:31:05,379 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-23) [3514bad5] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:31:05,397 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [3514bad5] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:31:15,451 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:31:15,482 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:31:15,484 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:31:15,501 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-60) START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 470d4d58 >2012-11-08 09:31:15,522 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-60) spmStart polling started: taskId = 0efc41a8-736d-4891-88ab-243e600d60b2 >2012-11-08 09:31:16,554 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-60) Failed in HSMGetTaskStatusVDS method >2012-11-08 09:31:16,554 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-60) Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:31:16,554 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-60) spmStart polling ended: taskId = 0efc41a8-736d-4891-88ab-243e600d60b2 task status = finished >2012-11-08 09:31:16,554 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-60) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:31:16,581 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-60) spmStart polling ended, spm status: Free >2012-11-08 09:31:16,599 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-60) START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=0efc41a8-736d-4891-88ab-243e600d60b2), log id: 751d1c23 >2012-11-08 09:31:16,616 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-60) FINISH, HSMClearTaskVDSCommand, log id: 751d1c23 >2012-11-08 09:31:16,616 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-60) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4a2279e6, log id: 470d4d58 >2012-11-08 09:31:16,618 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-60) [1d711110] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:31:16,636 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) [1d711110] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:31:26,700 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-75) hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:31:26,730 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-75) SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:31:26,732 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-75) starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:31:26,750 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-75) START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5b215c1d >2012-11-08 09:31:26,778 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-75) spmStart polling started: taskId = 0d927594-3710-4d0d-b268-6c5fa1faeeb6 >2012-11-08 09:31:27,817 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-75) Failed in HSMGetTaskStatusVDS method >2012-11-08 09:31:27,818 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-75) Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:31:27,818 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-75) spmStart polling ended: taskId = 0d927594-3710-4d0d-b268-6c5fa1faeeb6 task status = finished >2012-11-08 09:31:27,818 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-75) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:31:27,841 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-75) spmStart polling ended, spm status: Free >2012-11-08 09:31:27,854 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-75) START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=0d927594-3710-4d0d-b268-6c5fa1faeeb6), log id: 5065d0d >2012-11-08 09:31:27,877 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-75) FINISH, HSMClearTaskVDSCommand, log id: 5065d0d >2012-11-08 09:31:27,877 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-75) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@64c7030e, log id: 5b215c1d >2012-11-08 09:31:27,878 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-75) [4599bfd2] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:31:27,896 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-75) [4599bfd2] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:31:37,952 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-20) hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:31:37,981 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-20) SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:31:37,984 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-20) starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:31:38,000 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-20) START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 740bfb90 >2012-11-08 09:31:38,024 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-20) spmStart polling started: taskId = 6fde4e06-bf42-4aef-af17-66fb9ec12aa8 >2012-11-08 09:31:39,057 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-20) Failed in HSMGetTaskStatusVDS method >2012-11-08 09:31:39,058 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-20) Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:31:39,058 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-20) spmStart polling ended: taskId = 6fde4e06-bf42-4aef-af17-66fb9ec12aa8 task status = finished >2012-11-08 09:31:39,058 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-20) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:31:39,088 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-20) spmStart polling ended, spm status: Free >2012-11-08 09:31:39,103 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-20) START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=6fde4e06-bf42-4aef-af17-66fb9ec12aa8), log id: 617d90c3 >2012-11-08 09:31:39,121 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-20) FINISH, HSMClearTaskVDSCommand, log id: 617d90c3 >2012-11-08 09:31:39,121 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-20) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7aab2435, log id: 740bfb90 >2012-11-08 09:31:39,123 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-20) [79d6527d] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:31:39,144 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-20) [79d6527d] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:31:45,317 INFO [org.ovirt.engine.core.bll.SetVmTicketCommand] (ajp-/127.0.0.1:8702-8) [4681a0f7] Running command: SetVmTicketCommand internal: false. Entities affected : ID: e228f541-76b7-4a9b-b313-a6bacc2b73b9 Type: VM >2012-11-08 09:31:45,351 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp-/127.0.0.1:8702-8) [4681a0f7] START, SetVmTicketVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=e228f541-76b7-4a9b-b313-a6bacc2b73b9, ticket=J5CwUuosvlXC, validTime=120,m userName=admin@internal, userId=fdfc627c-d875-11e0-90f0-83df133b58cc), log id: 75e92b7b >2012-11-08 09:31:45,381 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp-/127.0.0.1:8702-8) [4681a0f7] FINISH, SetVmTicketVDSCommand, log id: 75e92b7b >2012-11-08 09:31:49,202 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-43) hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:31:49,230 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-43) SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:31:49,233 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-43) starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:31:49,249 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-43) START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7e23c115 >2012-11-08 09:31:49,271 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-43) spmStart polling started: taskId = 5ec11f3c-0d7a-4cb1-8675-4aae07a29e24 >2012-11-08 09:31:50,302 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-43) Failed in HSMGetTaskStatusVDS method >2012-11-08 09:31:50,302 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-43) Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:31:50,302 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-43) spmStart polling ended: taskId = 5ec11f3c-0d7a-4cb1-8675-4aae07a29e24 task status = finished >2012-11-08 09:31:50,302 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-43) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:31:50,326 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-43) spmStart polling ended, spm status: Free >2012-11-08 09:31:50,342 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-43) START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=5ec11f3c-0d7a-4cb1-8675-4aae07a29e24), log id: 74a82b5b >2012-11-08 09:31:50,365 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-43) FINISH, HSMClearTaskVDSCommand, log id: 74a82b5b >2012-11-08 09:31:50,365 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-43) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4cb69f32, log id: 7e23c115 >2012-11-08 09:31:50,367 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-43) [698db2f6] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:31:50,394 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-43) [698db2f6] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:31:56,226 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-50) [488d048b] START, SetVdsStatusVDSCommand(HostName = rhs-gp-srv12.lab.eng.blr.redhat.com, HostId = 056a9890-29af-11e2-b5a8-52540035ed48, status=NonResponsive, nonOperationalReason=NONE), log id: 4afca1fa >2012-11-08 09:31:56,231 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-50) [488d048b] FINISH, SetVdsStatusVDSCommand, log id: 4afca1fa >2012-11-08 09:31:57,236 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-71) [181586db] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:31:59,247 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-82) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:32:00,449 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-88) hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:32:00,478 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-88) SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:32:00,481 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-88) starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:32:00,498 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-88) START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 522fe76f >2012-11-08 09:32:00,519 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-88) spmStart polling started: taskId = 220378fa-258f-4dbc-92ea-ce63310d0107 >2012-11-08 09:32:01,256 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-86) [78cca392] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:32:01,551 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-88) Failed in HSMGetTaskStatusVDS method >2012-11-08 09:32:01,551 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-88) Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:32:01,551 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-88) spmStart polling ended: taskId = 220378fa-258f-4dbc-92ea-ce63310d0107 task status = finished >2012-11-08 09:32:01,551 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-88) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:32:01,579 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-88) spmStart polling ended, spm status: Free >2012-11-08 09:32:01,592 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-88) START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=220378fa-258f-4dbc-92ea-ce63310d0107), log id: 558556a3 >2012-11-08 09:32:01,610 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-88) FINISH, HSMClearTaskVDSCommand, log id: 558556a3 >2012-11-08 09:32:01,610 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-88) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@24fb1332, log id: 522fe76f >2012-11-08 09:32:01,612 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-88) [5fa94a95] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:32:01,634 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-88) [5fa94a95] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:32:03,265 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-3) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:32:05,275 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-62) [7d0da67e] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:32:07,285 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-99) [7f760eef] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:32:09,297 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-51) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:32:11,296 INFO [org.ovirt.engine.core.bll.MaintananceNumberOfVdssCommand] (pool-4-thread-50) [4ebee3f3] Running command: MaintananceNumberOfVdssCommand internal: false. Entities affected : ID: 056a9890-29af-11e2-b5a8-52540035ed48 Type: VDS >2012-11-08 09:32:11,305 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-18) [7ab5f38b] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:32:11,317 INFO [org.ovirt.engine.core.bll.MaintananceVdsCommand] (pool-4-thread-50) [4ebee3f3] Running command: MaintananceVdsCommand internal: true. Entities affected : ID: 056a9890-29af-11e2-b5a8-52540035ed48 Type: VDS >2012-11-08 09:32:11,320 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-50) [4ebee3f3] START, SetVdsStatusVDSCommand(HostName = rhs-gp-srv12.lab.eng.blr.redhat.com, HostId = 056a9890-29af-11e2-b5a8-52540035ed48, status=Maintenance, nonOperationalReason=NONE), log id: 2103a22b >2012-11-08 09:32:11,322 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-50) [4ebee3f3] FINISH, SetVdsStatusVDSCommand, log id: 2103a22b >2012-11-08 09:32:11,692 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-24) hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:32:11,718 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-24) SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:32:11,722 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-24) starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:32:11,739 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-24) START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3c567744 >2012-11-08 09:32:11,767 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-24) spmStart polling started: taskId = f7ef2495-482c-4adb-8ae5-df8ecbcc6926 >2012-11-08 09:32:12,807 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-24) Failed in HSMGetTaskStatusVDS method >2012-11-08 09:32:12,807 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-24) Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:32:12,807 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-24) spmStart polling ended: taskId = f7ef2495-482c-4adb-8ae5-df8ecbcc6926 task status = finished >2012-11-08 09:32:12,807 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-24) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:32:12,838 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-24) spmStart polling ended, spm status: Free >2012-11-08 09:32:12,851 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-24) START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=f7ef2495-482c-4adb-8ae5-df8ecbcc6926), log id: 4d87b07f >2012-11-08 09:32:12,874 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-24) FINISH, HSMClearTaskVDSCommand, log id: 4d87b07f >2012-11-08 09:32:12,874 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-24) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@754899d, log id: 3c567744 >2012-11-08 09:32:12,877 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-24) [3f16a62d] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:32:12,900 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-24) [3f16a62d] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:32:22,954 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-53) hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:32:22,982 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-53) SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:32:22,985 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-53) starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:32:23,005 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-53) START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2edcbe9d >2012-11-08 09:32:23,032 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-53) spmStart polling started: taskId = 16fa5ad6-c69a-4bea-991b-f45c2897a7ab >2012-11-08 09:32:23,265 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-47) [28e3a2cf] START, SetVdsStatusVDSCommand(HostName = rhs-gp-srv15.lab.eng.blr.redhat.com, HostId = 12e4f376-29af-11e2-b479-52540035ed48, status=NonResponsive, nonOperationalReason=NONE), log id: 1e664d74 >2012-11-08 09:32:23,270 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-47) [28e3a2cf] FINISH, SetVdsStatusVDSCommand, log id: 1e664d74 >2012-11-08 09:32:23,696 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-54) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:32:24,074 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-53) Failed in HSMGetTaskStatusVDS method >2012-11-08 09:32:24,074 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-53) Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:32:24,074 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-53) spmStart polling ended: taskId = 16fa5ad6-c69a-4bea-991b-f45c2897a7ab task status = finished >2012-11-08 09:32:24,074 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-53) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:32:24,101 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-53) spmStart polling ended, spm status: Free >2012-11-08 09:32:24,112 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-53) START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=16fa5ad6-c69a-4bea-991b-f45c2897a7ab), log id: 296c7c1 >2012-11-08 09:32:24,135 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-53) FINISH, HSMClearTaskVDSCommand, log id: 296c7c1 >2012-11-08 09:32:24,135 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-53) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7914513a, log id: 2edcbe9d >2012-11-08 09:32:24,137 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-53) [24e0f4f0] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:32:24,161 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-53) [24e0f4f0] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:32:24,894 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (ajp-/127.0.0.1:8702-2) Lock Acquired to object EngineLock [exclusiveLocks= key: 056a9890-29af-11e2-b5a8-52540035ed48 value: VDS >, sharedLocks= ] >2012-11-08 09:32:24,911 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (pool-4-thread-47) [244bce6f] Running command: ActivateVdsCommand internal: false. Entities affected : ID: 056a9890-29af-11e2-b5a8-52540035ed48 Type: VDS >2012-11-08 09:32:24,922 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-47) [244bce6f] START, SetVdsStatusVDSCommand(HostName = rhs-gp-srv12.lab.eng.blr.redhat.com, HostId = 056a9890-29af-11e2-b5a8-52540035ed48, status=Unassigned, nonOperationalReason=NONE), log id: 4684390e >2012-11-08 09:32:24,925 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-47) [244bce6f] FINISH, SetVdsStatusVDSCommand, log id: 4684390e >2012-11-08 09:32:24,928 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (pool-4-thread-47) [244bce6f] START, ActivateVdsVDSCommand(HostName = rhs-gp-srv12.lab.eng.blr.redhat.com, HostId = 056a9890-29af-11e2-b5a8-52540035ed48), log id: 1d00f531 >2012-11-08 09:32:24,932 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (pool-4-thread-47) [244bce6f] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:32:24,932 ERROR [org.ovirt.engine.core.vdsbroker.VdsManager] (pool-4-thread-47) [244bce6f] VDS::handleNetworkException Server failed to respond, vds_id = 056a9890-29af-11e2-b5a8-52540035ed48, vds_name = rhs-gp-srv12.lab.eng.blr.redhat.com, error = VDSNetworkException: >2012-11-08 09:32:24,949 INFO [org.ovirt.engine.core.bll.VdsEventListener] (pool-4-thread-50) [244bce6f] ResourceManager::vdsNotResponding entered for Host 056a9890-29af-11e2-b5a8-52540035ed48, 10.70.36.16 >2012-11-08 09:32:24,957 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (pool-4-thread-47) [244bce6f] ResourceManager::activateVds - failed to get VDS = 056a9890-29af-11e2-b5a8-52540035ed48 capabilities with error: VDSNetworkException: . >2012-11-08 09:32:24,957 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (pool-4-thread-47) [244bce6f] ResourceManager::activateVds - failed to activate VDS = 056a9890-29af-11e2-b5a8-52540035ed48 >2012-11-08 09:32:24,963 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (pool-4-thread-47) [11a3f5e5] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 056a9890-29af-11e2-b5a8-52540035ed48 Type: VDS >2012-11-08 09:32:24,964 WARN [org.ovirt.engine.core.bll.VdsNotRespondingTreatmentCommand] (pool-4-thread-50) [2febc41a] CanDoAction of action VdsNotRespondingTreatment failed. Reasons:VDS_FENCING_DISABLED >2012-11-08 09:32:24,967 WARN [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (pool-4-thread-47) [10487b07] CanDoAction of action HandleVdsVersion failed. Reasons:VDS_CANNOT_CHECK_VERSION_HOST_NON_RESPONSIVE >2012-11-08 09:32:24,967 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (pool-4-thread-47) [10487b07] FINISH, ActivateVdsVDSCommand, log id: 1d00f531 >2012-11-08 09:32:24,981 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (pool-4-thread-47) [10487b07] Lock freed to object EngineLock [exclusiveLocks= key: 056a9890-29af-11e2-b5a8-52540035ed48 value: VDS >, sharedLocks= ] >2012-11-08 09:32:25,326 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-72) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:32:25,707 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-55) [367e311c] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:32:27,337 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-46) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:32:27,719 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-56) [4783ef00] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:32:29,346 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-66) [65296545] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:32:29,731 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-70) [c5e7fc0] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:32:31,355 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-73) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:32:31,743 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-74) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:32:33,366 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-80) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:32:33,754 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-82) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:32:34,221 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:32:34,248 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:32:34,251 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:32:34,262 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-90) START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 18bbccdd >2012-11-08 09:32:34,284 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-90) spmStart polling started: taskId = b62b9c33-c7b7-4fef-80b2-da7c86bcad73 >2012-11-08 09:32:35,327 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-90) Failed in HSMGetTaskStatusVDS method >2012-11-08 09:32:35,327 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-90) Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:32:35,327 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-90) spmStart polling ended: taskId = b62b9c33-c7b7-4fef-80b2-da7c86bcad73 task status = finished >2012-11-08 09:32:35,328 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-90) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:32:35,359 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-90) spmStart polling ended, spm status: Free >2012-11-08 09:32:35,374 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-98) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:32:35,375 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-90) START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=b62b9c33-c7b7-4fef-80b2-da7c86bcad73), log id: b61e025 >2012-11-08 09:32:35,392 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-90) FINISH, HSMClearTaskVDSCommand, log id: b61e025 >2012-11-08 09:32:35,392 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-90) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7c6f0a79, log id: 18bbccdd >2012-11-08 09:32:35,394 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-90) [5e18cdc1] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:32:35,408 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [5e18cdc1] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:32:35,762 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-86) [78cca392] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:32:37,384 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-87) [3089800c] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:32:37,773 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-3) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:32:39,394 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-97) [7ad09500] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:32:39,785 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-96) [88cae39] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:32:41,406 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-10) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:32:41,809 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-14) [89ac041] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:32:43,415 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-17) [36467d8f] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:32:43,821 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-51) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:32:45,425 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-16) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:32:45,461 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:32:45,488 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:32:45,490 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:32:45,507 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 73c0c840 >2012-11-08 09:32:45,529 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) spmStart polling started: taskId = 086f329f-2b36-472c-8bb6-5a89b2b2538a >2012-11-08 09:32:45,830 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-18) [7ab5f38b] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:32:46,559 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-7) Failed in HSMGetTaskStatusVDS method >2012-11-08 09:32:46,559 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-7) Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:32:46,559 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) spmStart polling ended: taskId = 086f329f-2b36-472c-8bb6-5a89b2b2538a task status = finished >2012-11-08 09:32:46,559 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:32:46,586 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) spmStart polling ended, spm status: Free >2012-11-08 09:32:46,604 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-7) START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=086f329f-2b36-472c-8bb6-5a89b2b2538a), log id: 76978239 >2012-11-08 09:32:46,621 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-7) FINISH, HSMClearTaskVDSCommand, log id: 76978239 >2012-11-08 09:32:46,621 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@54e6a902, log id: 73c0c840 >2012-11-08 09:32:46,623 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-7) [7361edd9] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:32:46,649 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [7361edd9] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:32:47,437 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-30) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:32:47,840 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-25) [23551d18] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:32:49,454 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-22) [5fe2db58] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:32:49,850 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-34) [7d41af0b] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:32:51,465 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-26) [733c0daa] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:32:51,857 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-39) [6878973a] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:32:53,474 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-29) [745c1fd1] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:32:53,866 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-32) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:32:55,487 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-40) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:32:55,876 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-38) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:32:56,705 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-52) hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:32:56,735 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-52) SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:32:56,738 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-52) starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:32:56,756 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-52) START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 29bd9cc3 >2012-11-08 09:32:56,785 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-52) spmStart polling started: taskId = dc60d7d5-dc92-4489-9ae0-be2e8e257647 >2012-11-08 09:32:57,498 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-94) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:32:57,822 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-52) Failed in HSMGetTaskStatusVDS method >2012-11-08 09:32:57,822 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-52) Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:32:57,822 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-52) spmStart polling ended: taskId = dc60d7d5-dc92-4489-9ae0-be2e8e257647 task status = finished >2012-11-08 09:32:57,822 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-52) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:32:57,854 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-52) spmStart polling ended, spm status: Free >2012-11-08 09:32:57,868 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-52) START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=dc60d7d5-dc92-4489-9ae0-be2e8e257647), log id: 78b8e28d >2012-11-08 09:32:57,883 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-54) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:32:57,888 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-52) FINISH, HSMClearTaskVDSCommand, log id: 78b8e28d >2012-11-08 09:32:57,888 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-52) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6f8c4bb0, log id: 29bd9cc3 >2012-11-08 09:32:57,890 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-52) [63bd4906] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:32:57,923 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-52) [63bd4906] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:32:59,510 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-60) [1d711110] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:32:59,894 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-55) [367e311c] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:33:01,521 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-46) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:33:01,906 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-68) [77dbb526] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:33:03,533 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-63) [4ee90ab8] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:33:03,916 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-66) [65296545] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:33:05,544 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-65) [2006b9d4] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:33:05,925 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-35) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:33:06,085 INFO [org.ovirt.engine.core.bll.MaintananceNumberOfVdssCommand] (pool-4-thread-47) [792d95b3] Running command: MaintananceNumberOfVdssCommand internal: false. Entities affected : ID: 056a9890-29af-11e2-b5a8-52540035ed48 Type: VDS >2012-11-08 09:33:06,104 INFO [org.ovirt.engine.core.bll.MaintananceVdsCommand] (pool-4-thread-47) [792d95b3] Running command: MaintananceVdsCommand internal: true. Entities affected : ID: 056a9890-29af-11e2-b5a8-52540035ed48 Type: VDS >2012-11-08 09:33:06,108 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-47) [792d95b3] START, SetVdsStatusVDSCommand(HostName = rhs-gp-srv12.lab.eng.blr.redhat.com, HostId = 056a9890-29af-11e2-b5a8-52540035ed48, status=Maintenance, nonOperationalReason=NONE), log id: 1f0734e6 >2012-11-08 09:33:06,111 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-47) [792d95b3] FINISH, SetVdsStatusVDSCommand, log id: 1f0734e6 >2012-11-08 09:33:07,935 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-80) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:33:07,985 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:33:08,015 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:33:08,018 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:33:08,035 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-77) START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3b1500ce >2012-11-08 09:33:08,058 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-77) spmStart polling started: taskId = 923d1d21-bc54-4c16-8885-f5e07ec55adc >2012-11-08 09:33:09,088 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-77) Failed in HSMGetTaskStatusVDS method >2012-11-08 09:33:09,088 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-77) Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:33:09,088 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-77) spmStart polling ended: taskId = 923d1d21-bc54-4c16-8885-f5e07ec55adc task status = finished >2012-11-08 09:33:09,088 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-77) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:33:09,117 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-77) spmStart polling ended, spm status: Free >2012-11-08 09:33:09,127 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-77) START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=923d1d21-bc54-4c16-8885-f5e07ec55adc), log id: 6260c27a >2012-11-08 09:33:09,147 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-77) FINISH, HSMClearTaskVDSCommand, log id: 6260c27a >2012-11-08 09:33:09,147 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-77) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3e800a0b, log id: 3b1500ce >2012-11-08 09:33:09,148 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-77) [4813783b] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:33:09,171 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [4813783b] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:33:09,197 INFO [org.ovirt.engine.core.bll.MaintananceNumberOfVdssCommand] (pool-4-thread-45) [6eb00e7] Running command: MaintananceNumberOfVdssCommand internal: false. Entities affected : ID: 12e4f376-29af-11e2-b479-52540035ed48 Type: VDS >2012-11-08 09:33:09,206 INFO [org.ovirt.engine.core.bll.MaintananceVdsCommand] (pool-4-thread-45) [6eb00e7] Running command: MaintananceVdsCommand internal: true. Entities affected : ID: 12e4f376-29af-11e2-b479-52540035ed48 Type: VDS >2012-11-08 09:33:09,208 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-45) [6eb00e7] START, SetVdsStatusVDSCommand(HostName = rhs-gp-srv15.lab.eng.blr.redhat.com, HostId = 12e4f376-29af-11e2-b479-52540035ed48, status=Maintenance, nonOperationalReason=NONE), log id: 3e8f57aa >2012-11-08 09:33:09,209 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-45) [6eb00e7] FINISH, SetVdsStatusVDSCommand, log id: 3e8f57aa >2012-11-08 09:33:19,228 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-4) hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:33:19,256 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-4) SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:33:19,258 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-4) starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:33:19,275 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-4) START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 238e5dfc >2012-11-08 09:33:19,304 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-4) spmStart polling started: taskId = 8eeae9dc-8286-4ded-98f8-fe7b4cdbcaf4 >2012-11-08 09:33:20,344 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-4) Failed in HSMGetTaskStatusVDS method >2012-11-08 09:33:20,344 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-4) Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:33:20,344 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-4) spmStart polling ended: taskId = 8eeae9dc-8286-4ded-98f8-fe7b4cdbcaf4 task status = finished >2012-11-08 09:33:20,344 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-4) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:33:20,375 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-4) spmStart polling ended, spm status: Free >2012-11-08 09:33:20,389 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-4) START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=8eeae9dc-8286-4ded-98f8-fe7b4cdbcaf4), log id: 21157bd1 >2012-11-08 09:33:20,406 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-4) FINISH, HSMClearTaskVDSCommand, log id: 21157bd1 >2012-11-08 09:33:20,407 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-4) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@973e74f, log id: 238e5dfc >2012-11-08 09:33:20,409 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-4) [1856f862] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:33:20,430 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-4) [1856f862] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:33:30,489 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-49) hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:33:30,516 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-49) SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:33:30,518 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-49) starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:33:30,546 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-49) START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 277ec0a4 >2012-11-08 09:33:30,570 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-49) spmStart polling started: taskId = a6f12fa9-662e-4444-8a6e-1b95a0d865bc >2012-11-08 09:33:31,615 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-49) Failed in HSMGetTaskStatusVDS method >2012-11-08 09:33:31,616 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-49) Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:33:31,616 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-49) spmStart polling ended: taskId = a6f12fa9-662e-4444-8a6e-1b95a0d865bc task status = finished >2012-11-08 09:33:31,616 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-49) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:33:31,645 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-49) spmStart polling ended, spm status: Free >2012-11-08 09:33:31,663 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-49) START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=a6f12fa9-662e-4444-8a6e-1b95a0d865bc), log id: 389939f7 >2012-11-08 09:33:31,688 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-49) FINISH, HSMClearTaskVDSCommand, log id: 389939f7 >2012-11-08 09:33:31,688 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-49) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@53aca428, log id: 277ec0a4 >2012-11-08 09:33:31,690 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-49) [6b4d6902] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:33:31,707 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-49) [6b4d6902] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:33:39,307 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-76) Failed to decrypt Data must not be longer than 256 bytes >2012-11-08 09:33:39,318 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-70) Failed to decrypt Data must start with zero >2012-11-08 09:33:41,766 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-80) hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:33:41,790 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-80) SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:33:41,792 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-80) starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:33:41,811 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-80) START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6542a9bd >2012-11-08 09:33:41,836 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-80) spmStart polling started: taskId = c44845f3-7dd9-4431-87b4-bc881c6a68fa >2012-11-08 09:33:42,870 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-80) Failed in HSMGetTaskStatusVDS method >2012-11-08 09:33:42,871 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-80) Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:33:42,871 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-80) spmStart polling ended: taskId = c44845f3-7dd9-4431-87b4-bc881c6a68fa task status = finished >2012-11-08 09:33:42,871 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-80) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:33:42,897 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-80) spmStart polling ended, spm status: Free >2012-11-08 09:33:42,909 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-80) START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=c44845f3-7dd9-4431-87b4-bc881c6a68fa), log id: 7f357e1e >2012-11-08 09:33:42,931 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-80) FINISH, HSMClearTaskVDSCommand, log id: 7f357e1e >2012-11-08 09:33:42,931 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-80) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@10eb24d1, log id: 6542a9bd >2012-11-08 09:33:42,933 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-80) [3dbe1e04] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:33:42,951 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-80) [3dbe1e04] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:33:53,007 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:33:53,034 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:33:53,036 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:33:53,053 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-51) START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 76128ca5 >2012-11-08 09:33:53,078 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-51) spmStart polling started: taskId = 61c5dbd8-d13a-4886-8e5d-c16867dd2520 >2012-11-08 09:33:54,111 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-51) Failed in HSMGetTaskStatusVDS method >2012-11-08 09:33:54,111 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-51) Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:33:54,111 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-51) spmStart polling ended: taskId = 61c5dbd8-d13a-4886-8e5d-c16867dd2520 task status = finished >2012-11-08 09:33:54,111 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-51) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:33:54,137 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-51) spmStart polling ended, spm status: Free >2012-11-08 09:33:54,149 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-51) START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=61c5dbd8-d13a-4886-8e5d-c16867dd2520), log id: 1b9356f3 >2012-11-08 09:33:54,170 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-51) FINISH, HSMClearTaskVDSCommand, log id: 1b9356f3 >2012-11-08 09:33:54,171 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-51) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@663179a9, log id: 76128ca5 >2012-11-08 09:33:54,172 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-51) [3a2a7aa9] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:33:54,195 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) [3a2a7aa9] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:34:04,247 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-45) hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:34:04,274 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-45) SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:34:04,276 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-45) starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:34:04,293 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-45) START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7960b373 >2012-11-08 09:34:04,323 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-45) spmStart polling started: taskId = 86ebc797-6306-4394-9d90-1bb69fe97d0a >2012-11-08 09:34:05,367 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-45) Failed in HSMGetTaskStatusVDS method >2012-11-08 09:34:05,367 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-45) Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:34:05,367 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-45) spmStart polling ended: taskId = 86ebc797-6306-4394-9d90-1bb69fe97d0a task status = finished >2012-11-08 09:34:05,367 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-45) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:34:05,395 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-45) spmStart polling ended, spm status: Free >2012-11-08 09:34:05,412 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-45) START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=86ebc797-6306-4394-9d90-1bb69fe97d0a), log id: 3b448ac2 >2012-11-08 09:34:05,437 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-45) FINISH, HSMClearTaskVDSCommand, log id: 3b448ac2 >2012-11-08 09:34:05,437 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-45) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@10f1e4f4, log id: 7960b373 >2012-11-08 09:34:05,438 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-45) [78ee57dc] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:34:05,467 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-45) [78ee57dc] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:34:15,528 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [181586db] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:34:15,555 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [181586db] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:34:15,558 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [181586db] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:34:15,573 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-71) [181586db] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 103a13e1 >2012-11-08 09:34:15,597 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-71) [181586db] spmStart polling started: taskId = 87817458-a393-4f64-b861-b79ca9dacd58 >2012-11-08 09:34:16,626 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-71) [181586db] Failed in HSMGetTaskStatusVDS method >2012-11-08 09:34:16,626 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-71) [181586db] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:34:16,626 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-71) [181586db] spmStart polling ended: taskId = 87817458-a393-4f64-b861-b79ca9dacd58 task status = finished >2012-11-08 09:34:16,626 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-71) [181586db] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:34:16,654 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-71) [181586db] spmStart polling ended, spm status: Free >2012-11-08 09:34:16,666 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-71) [181586db] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=87817458-a393-4f64-b861-b79ca9dacd58), log id: 24b719cc >2012-11-08 09:34:16,684 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-71) [181586db] FINISH, HSMClearTaskVDSCommand, log id: 24b719cc >2012-11-08 09:34:16,684 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-71) [181586db] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@70feb2e8, log id: 103a13e1 >2012-11-08 09:34:16,686 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-71) [6193d74c] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:34:16,707 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [6193d74c] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:34:26,772 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:34:26,796 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:34:26,798 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:34:26,813 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-2) START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3643d7dd >2012-11-08 09:34:26,835 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-2) spmStart polling started: taskId = 61af73ef-6957-473a-a520-53638c0e25d7 >2012-11-08 09:34:27,869 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-2) Failed in HSMGetTaskStatusVDS method >2012-11-08 09:34:27,870 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-2) Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:34:27,870 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-2) spmStart polling ended: taskId = 61af73ef-6957-473a-a520-53638c0e25d7 task status = finished >2012-11-08 09:34:27,870 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-2) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:34:27,901 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-2) spmStart polling ended, spm status: Free >2012-11-08 09:34:27,920 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-2) START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=61af73ef-6957-473a-a520-53638c0e25d7), log id: 705133b >2012-11-08 09:34:27,938 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-2) FINISH, HSMClearTaskVDSCommand, log id: 705133b >2012-11-08 09:34:27,938 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-2) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@f5b5d2f, log id: 3643d7dd >2012-11-08 09:34:27,940 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-2) [3742c089] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:34:27,960 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) [3742c089] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:34:38,017 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-40) hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:34:38,043 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-40) SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:34:38,045 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-40) starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:34:38,062 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-40) START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5c4993ae >2012-11-08 09:34:38,089 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-40) spmStart polling started: taskId = d0ef9a24-20d9-4176-83dc-1ebce51defcb >2012-11-08 09:34:39,134 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-40) Failed in HSMGetTaskStatusVDS method >2012-11-08 09:34:39,134 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-40) Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:34:39,134 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-40) spmStart polling ended: taskId = d0ef9a24-20d9-4176-83dc-1ebce51defcb task status = finished >2012-11-08 09:34:39,134 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-40) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:34:39,161 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-40) spmStart polling ended, spm status: Free >2012-11-08 09:34:39,174 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-40) START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=d0ef9a24-20d9-4176-83dc-1ebce51defcb), log id: 164789e5 >2012-11-08 09:34:39,192 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-40) FINISH, HSMClearTaskVDSCommand, log id: 164789e5 >2012-11-08 09:34:39,192 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-40) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@32ae24d0, log id: 5c4993ae >2012-11-08 09:34:39,195 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-40) [46f2edae] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:34:39,215 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-40) [46f2edae] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:34:49,291 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-35) hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:34:49,318 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-35) SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:34:49,320 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-35) starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:34:49,335 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-35) START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 56f13b54 >2012-11-08 09:34:49,365 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-35) spmStart polling started: taskId = 1a14ba37-2c3c-44cf-aa0c-75b8ecbb4f51 >2012-11-08 09:34:50,399 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-35) Failed in HSMGetTaskStatusVDS method >2012-11-08 09:34:50,399 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-35) Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:34:50,399 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-35) spmStart polling ended: taskId = 1a14ba37-2c3c-44cf-aa0c-75b8ecbb4f51 task status = finished >2012-11-08 09:34:50,399 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-35) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:34:50,430 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-35) spmStart polling ended, spm status: Free >2012-11-08 09:34:50,443 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-35) START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=1a14ba37-2c3c-44cf-aa0c-75b8ecbb4f51), log id: 787ce6f7 >2012-11-08 09:34:50,464 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-35) FINISH, HSMClearTaskVDSCommand, log id: 787ce6f7 >2012-11-08 09:34:50,464 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-35) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@501f9fed, log id: 56f13b54 >2012-11-08 09:34:50,468 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-35) [4c3ca9b3] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:34:50,491 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-35) [4c3ca9b3] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:35:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-8) Autorecovering hosts is disabled, skipping >2012-11-08 09:35:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-8) Autorecovering storage domains is disabled, skipping >2012-11-08 09:35:00,552 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-11) hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:35:00,577 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-11) SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:35:00,579 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-11) starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:35:00,590 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-11) START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 61e61e5f >2012-11-08 09:35:00,619 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-11) spmStart polling started: taskId = ca552204-bbce-40e6-bd68-e32eb000c7e3 >2012-11-08 09:35:01,657 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-11) Failed in HSMGetTaskStatusVDS method >2012-11-08 09:35:01,658 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-11) Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:35:01,658 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-11) spmStart polling ended: taskId = ca552204-bbce-40e6-bd68-e32eb000c7e3 task status = finished >2012-11-08 09:35:01,658 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-11) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:35:01,690 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-11) spmStart polling ended, spm status: Free >2012-11-08 09:35:01,708 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-11) START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=ca552204-bbce-40e6-bd68-e32eb000c7e3), log id: 4c6409da >2012-11-08 09:35:01,737 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-11) FINISH, HSMClearTaskVDSCommand, log id: 4c6409da >2012-11-08 09:35:01,737 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-11) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@f7857db, log id: 61e61e5f >2012-11-08 09:35:01,740 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-11) [25cff2cc] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:35:01,763 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-11) [25cff2cc] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:35:11,822 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:35:11,849 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:35:11,853 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:35:11,869 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-38) START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 36096c1f >2012-11-08 09:35:11,899 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-38) spmStart polling started: taskId = 535368c1-237e-4ddf-b6d3-dbe4cd155a81 >2012-11-08 09:35:12,935 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-38) Failed in HSMGetTaskStatusVDS method >2012-11-08 09:35:12,935 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-38) Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:35:12,935 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-38) spmStart polling ended: taskId = 535368c1-237e-4ddf-b6d3-dbe4cd155a81 task status = finished >2012-11-08 09:35:12,935 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-38) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:35:12,962 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-38) spmStart polling ended, spm status: Free >2012-11-08 09:35:12,974 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-38) START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=535368c1-237e-4ddf-b6d3-dbe4cd155a81), log id: 662b4679 >2012-11-08 09:35:12,996 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-38) FINISH, HSMClearTaskVDSCommand, log id: 662b4679 >2012-11-08 09:35:12,996 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-38) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@231fe656, log id: 36096c1f >2012-11-08 09:35:12,999 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-38) [34b2a1e] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:35:13,024 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [34b2a1e] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:35:23,081 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:35:23,109 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:35:23,111 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:35:23,129 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6be0a59a >2012-11-08 09:35:23,151 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) spmStart polling started: taskId = 64b9f776-28d8-43ed-b99d-69c25c9f5ead >2012-11-08 09:35:24,182 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-85) Failed in HSMGetTaskStatusVDS method >2012-11-08 09:35:24,182 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-85) Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:35:24,182 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) spmStart polling ended: taskId = 64b9f776-28d8-43ed-b99d-69c25c9f5ead task status = finished >2012-11-08 09:35:24,182 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:35:24,211 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) spmStart polling ended, spm status: Free >2012-11-08 09:35:24,228 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-85) START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=64b9f776-28d8-43ed-b99d-69c25c9f5ead), log id: 2145091f >2012-11-08 09:35:24,245 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-85) FINISH, HSMClearTaskVDSCommand, log id: 2145091f >2012-11-08 09:35:24,245 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3b59e349, log id: 6be0a59a >2012-11-08 09:35:24,247 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-85) [a58ec10] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:35:24,272 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [a58ec10] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:35:29,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (ajp-/127.0.0.1:8702-10) Lock Acquired to object EngineLock [exclusiveLocks= key: 12e4f376-29af-11e2-b479-52540035ed48 value: VDS >, sharedLocks= ] >2012-11-08 09:35:29,018 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (pool-4-thread-45) [fbed81] Running command: ActivateVdsCommand internal: false. Entities affected : ID: 12e4f376-29af-11e2-b479-52540035ed48 Type: VDS >2012-11-08 09:35:29,031 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-45) [fbed81] START, SetVdsStatusVDSCommand(HostName = rhs-gp-srv15.lab.eng.blr.redhat.com, HostId = 12e4f376-29af-11e2-b479-52540035ed48, status=Unassigned, nonOperationalReason=NONE), log id: 501bbcb5 >2012-11-08 09:35:29,033 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-45) [fbed81] FINISH, SetVdsStatusVDSCommand, log id: 501bbcb5 >2012-11-08 09:35:29,035 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (pool-4-thread-45) [fbed81] START, ActivateVdsVDSCommand(HostName = rhs-gp-srv15.lab.eng.blr.redhat.com, HostId = 12e4f376-29af-11e2-b479-52540035ed48), log id: 4c22505 >2012-11-08 09:35:29,038 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (pool-4-thread-45) [fbed81] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:35:29,038 ERROR [org.ovirt.engine.core.vdsbroker.VdsManager] (pool-4-thread-45) [fbed81] VDS::handleNetworkException Server failed to respond, vds_id = 12e4f376-29af-11e2-b479-52540035ed48, vds_name = rhs-gp-srv15.lab.eng.blr.redhat.com, error = VDSNetworkException: >2012-11-08 09:35:29,046 INFO [org.ovirt.engine.core.bll.VdsEventListener] (pool-4-thread-50) [fbed81] ResourceManager::vdsNotResponding entered for Host 12e4f376-29af-11e2-b479-52540035ed48, 10.70.36.19 >2012-11-08 09:35:29,048 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (pool-4-thread-45) [fbed81] ResourceManager::activateVds - failed to get VDS = 12e4f376-29af-11e2-b479-52540035ed48 capabilities with error: VDSNetworkException: . >2012-11-08 09:35:29,048 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (pool-4-thread-45) [fbed81] ResourceManager::activateVds - failed to activate VDS = 12e4f376-29af-11e2-b479-52540035ed48 >2012-11-08 09:35:29,052 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (pool-4-thread-45) [781dc89a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 12e4f376-29af-11e2-b479-52540035ed48 Type: VDS >2012-11-08 09:35:29,054 WARN [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (pool-4-thread-45) [674de349] CanDoAction of action HandleVdsVersion failed. Reasons:VDS_CANNOT_CHECK_VERSION_HOST_NON_RESPONSIVE >2012-11-08 09:35:29,054 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (pool-4-thread-45) [674de349] FINISH, ActivateVdsVDSCommand, log id: 4c22505 >2012-11-08 09:35:29,057 WARN [org.ovirt.engine.core.bll.VdsNotRespondingTreatmentCommand] (pool-4-thread-50) [7d5b3120] CanDoAction of action VdsNotRespondingTreatment failed. Reasons:VDS_FENCING_DISABLED >2012-11-08 09:35:29,070 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (pool-4-thread-45) [674de349] Lock freed to object EngineLock [exclusiveLocks= key: 12e4f376-29af-11e2-b479-52540035ed48 value: VDS >, sharedLocks= ] >2012-11-08 09:35:30,067 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-84) [642523c8] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:35:32,077 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-95) [41e3afd7] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:35:34,087 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-14) [89ac041] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:35:34,340 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:35:34,368 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:35:34,373 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:35:34,389 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 621d981d >2012-11-08 09:35:34,418 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) spmStart polling started: taskId = 79ef31e6-68b9-46f5-916a-9268954646d1 >2012-11-08 09:35:35,463 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-8) Failed in HSMGetTaskStatusVDS method >2012-11-08 09:35:35,463 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-8) Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:35:35,463 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) spmStart polling ended: taskId = 79ef31e6-68b9-46f5-916a-9268954646d1 task status = finished >2012-11-08 09:35:35,463 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:35:35,491 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) spmStart polling ended, spm status: Free >2012-11-08 09:35:35,503 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-8) START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=79ef31e6-68b9-46f5-916a-9268954646d1), log id: 634e603b >2012-11-08 09:35:35,520 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-8) FINISH, HSMClearTaskVDSCommand, log id: 634e603b >2012-11-08 09:35:35,521 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2e8ff46c, log id: 621d981d >2012-11-08 09:35:35,522 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-8) [6cf7f655] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:35:35,545 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [6cf7f655] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:35:36,097 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-18) [7ab5f38b] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:35:38,107 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-15) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:35:40,119 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-92) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:35:42,128 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-9) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:35:44,140 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-36) [6a24e609] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:35:45,601 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-44) hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:35:45,628 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-44) SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:35:45,631 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-44) starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:35:45,648 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-44) START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 151f356f >2012-11-08 09:35:45,677 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-44) spmStart polling started: taskId = 69591a51-d214-432d-aae4-82d596e121aa >2012-11-08 09:35:46,151 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-48) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:35:46,714 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-44) Failed in HSMGetTaskStatusVDS method >2012-11-08 09:35:46,715 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-44) Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:35:46,715 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-44) spmStart polling ended: taskId = 69591a51-d214-432d-aae4-82d596e121aa task status = finished >2012-11-08 09:35:46,715 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-44) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:35:46,741 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-44) spmStart polling ended, spm status: Free >2012-11-08 09:35:46,758 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-44) START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=69591a51-d214-432d-aae4-82d596e121aa), log id: 5db5346c >2012-11-08 09:35:46,783 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-44) FINISH, HSMClearTaskVDSCommand, log id: 5db5346c >2012-11-08 09:35:46,783 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-44) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2384ce84, log id: 151f356f >2012-11-08 09:35:46,785 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-44) [64124e53] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:35:46,801 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-44) [64124e53] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:35:48,158 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-38) [34b2a1e] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:35:50,168 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-43) [698db2f6] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:35:52,178 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-58) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:35:54,186 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-68) [77dbb526] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:35:56,196 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-93) [51520066] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:35:56,862 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) [2006b9d4] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:35:56,889 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) [2006b9d4] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:35:56,891 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) [2006b9d4] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:35:56,907 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-65) [2006b9d4] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 71405a6 >2012-11-08 09:35:56,929 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-65) [2006b9d4] spmStart polling started: taskId = 7e90198e-1d64-4e98-be5f-c095de504436 >2012-11-08 09:35:57,959 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-65) [2006b9d4] Failed in HSMGetTaskStatusVDS method >2012-11-08 09:35:57,960 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-65) [2006b9d4] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:35:57,960 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-65) [2006b9d4] spmStart polling ended: taskId = 7e90198e-1d64-4e98-be5f-c095de504436 task status = finished >2012-11-08 09:35:57,960 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-65) [2006b9d4] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:35:57,988 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-65) [2006b9d4] spmStart polling ended, spm status: Free >2012-11-08 09:35:58,001 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-65) [2006b9d4] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=7e90198e-1d64-4e98-be5f-c095de504436), log id: 7c891704 >2012-11-08 09:35:58,019 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-65) [2006b9d4] FINISH, HSMClearTaskVDSCommand, log id: 7c891704 >2012-11-08 09:35:58,019 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-65) [2006b9d4] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@c46477c, log id: 71405a6 >2012-11-08 09:35:58,021 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-65) [59ff242f] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:35:58,036 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) [59ff242f] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:35:58,204 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-35) [4c3ca9b3] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:36:00,213 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-77) [4813783b] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:36:02,224 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-91) [5f904976] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:36:04,234 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-84) [642523c8] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:36:06,244 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-5) [431362fc] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:36:08,085 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-6) hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:36:08,114 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-6) SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:36:08,117 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-6) starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:36:08,135 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-6) START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7d3359a >2012-11-08 09:36:08,156 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-6) spmStart polling started: taskId = 906bba72-0e97-49cb-a6ac-0c1e2eb58463 >2012-11-08 09:36:08,254 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-14) [89ac041] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:36:09,189 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-6) Failed in HSMGetTaskStatusVDS method >2012-11-08 09:36:09,189 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-6) Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:36:09,189 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-6) spmStart polling ended: taskId = 906bba72-0e97-49cb-a6ac-0c1e2eb58463 task status = finished >2012-11-08 09:36:09,189 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-6) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:36:09,218 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-6) spmStart polling ended, spm status: Free >2012-11-08 09:36:09,236 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-6) START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=906bba72-0e97-49cb-a6ac-0c1e2eb58463), log id: 5e57bca4 >2012-11-08 09:36:09,253 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-6) FINISH, HSMClearTaskVDSCommand, log id: 5e57bca4 >2012-11-08 09:36:09,253 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-6) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@5221e35f, log id: 7d3359a >2012-11-08 09:36:09,255 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-6) [2ed69c19] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:36:09,278 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-6) [2ed69c19] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:36:10,264 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-51) [3a2a7aa9] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:36:12,275 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-30) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:36:14,286 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-92) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:36:16,295 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-34) [7d41af0b] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:36:18,305 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-39) [6878973a] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:36:19,329 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-32) hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:36:19,357 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-32) SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:36:19,359 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-32) starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:36:19,376 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-32) START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 399e80b9 >2012-11-08 09:36:19,400 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-32) spmStart polling started: taskId = 05cab3b9-1a78-4fae-819d-9f144c4dd872 >2012-11-08 09:36:20,330 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-44) [64124e53] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:36:20,434 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-32) Failed in HSMGetTaskStatusVDS method >2012-11-08 09:36:20,434 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-32) Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:36:20,434 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-32) spmStart polling ended: taskId = 05cab3b9-1a78-4fae-819d-9f144c4dd872 task status = finished >2012-11-08 09:36:20,434 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-32) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:36:20,460 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-32) spmStart polling ended, spm status: Free >2012-11-08 09:36:20,472 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-32) START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=05cab3b9-1a78-4fae-819d-9f144c4dd872), log id: 26a1212d >2012-11-08 09:36:20,489 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-32) FINISH, HSMClearTaskVDSCommand, log id: 26a1212d >2012-11-08 09:36:20,489 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-32) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1c89252b, log id: 399e80b9 >2012-11-08 09:36:20,491 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-32) [6fe738af] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:36:20,519 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-32) [6fe738af] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:36:22,340 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-47) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:36:24,347 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-43) [698db2f6] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:36:25,148 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (ajp-/127.0.0.1:8702-11) Lock Acquired to object EngineLock [exclusiveLocks= key: 056a9890-29af-11e2-b5a8-52540035ed48 value: VDS >, sharedLocks= ] >2012-11-08 09:36:25,163 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (pool-4-thread-50) [520a8b63] Running command: ActivateVdsCommand internal: false. Entities affected : ID: 056a9890-29af-11e2-b5a8-52540035ed48 Type: VDS >2012-11-08 09:36:25,176 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-50) [520a8b63] START, SetVdsStatusVDSCommand(HostName = rhs-gp-srv12.lab.eng.blr.redhat.com, HostId = 056a9890-29af-11e2-b5a8-52540035ed48, status=Unassigned, nonOperationalReason=NONE), log id: 641ba409 >2012-11-08 09:36:25,178 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-50) [520a8b63] FINISH, SetVdsStatusVDSCommand, log id: 641ba409 >2012-11-08 09:36:25,180 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (pool-4-thread-50) [520a8b63] START, ActivateVdsVDSCommand(HostName = rhs-gp-srv12.lab.eng.blr.redhat.com, HostId = 056a9890-29af-11e2-b5a8-52540035ed48), log id: 551b055a >2012-11-08 09:36:25,183 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (pool-4-thread-50) [520a8b63] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:36:25,183 ERROR [org.ovirt.engine.core.vdsbroker.VdsManager] (pool-4-thread-50) [520a8b63] VDS::handleNetworkException Server failed to respond, vds_id = 056a9890-29af-11e2-b5a8-52540035ed48, vds_name = rhs-gp-srv12.lab.eng.blr.redhat.com, error = VDSNetworkException: >2012-11-08 09:36:25,198 INFO [org.ovirt.engine.core.bll.VdsEventListener] (pool-4-thread-45) [520a8b63] ResourceManager::vdsNotResponding entered for Host 056a9890-29af-11e2-b5a8-52540035ed48, 10.70.36.16 >2012-11-08 09:36:25,202 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (pool-4-thread-50) [520a8b63] ResourceManager::activateVds - failed to get VDS = 056a9890-29af-11e2-b5a8-52540035ed48 capabilities with error: VDSNetworkException: . >2012-11-08 09:36:25,202 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (pool-4-thread-50) [520a8b63] ResourceManager::activateVds - failed to activate VDS = 056a9890-29af-11e2-b5a8-52540035ed48 >2012-11-08 09:36:25,206 WARN [org.ovirt.engine.core.bll.VdsNotRespondingTreatmentCommand] (pool-4-thread-45) [4a62e0e3] CanDoAction of action VdsNotRespondingTreatment failed. Reasons:VDS_FENCING_DISABLED >2012-11-08 09:36:25,207 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (pool-4-thread-50) [412ca898] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 056a9890-29af-11e2-b5a8-52540035ed48 Type: VDS >2012-11-08 09:36:25,211 WARN [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (pool-4-thread-50) [72cb4b8e] CanDoAction of action HandleVdsVersion failed. Reasons:VDS_CANNOT_CHECK_VERSION_HOST_NON_RESPONSIVE >2012-11-08 09:36:25,211 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (pool-4-thread-50) [72cb4b8e] FINISH, ActivateVdsVDSCommand, log id: 551b055a >2012-11-08 09:36:25,230 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (pool-4-thread-50) [72cb4b8e] Lock freed to object EngineLock [exclusiveLocks= key: 056a9890-29af-11e2-b5a8-52540035ed48 value: VDS >, sharedLocks= ] >2012-11-08 09:36:25,736 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-60) [1d711110] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:36:26,351 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-57) [709f6bac] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:36:27,746 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-50) [501c9b34] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:36:28,377 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-68) [77dbb526] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:36:29,755 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-93) [51520066] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:36:30,386 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-70) [c5e7fc0] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:36:30,577 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:36:30,605 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:36:30,608 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:36:30,625 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-74) START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 1205e8bb >2012-11-08 09:36:30,654 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-74) spmStart polling started: taskId = 2a992655-221e-4492-812a-b1d978389a9b >2012-11-08 09:36:31,692 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-74) Failed in HSMGetTaskStatusVDS method >2012-11-08 09:36:31,692 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-74) Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:36:31,692 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-74) spmStart polling ended: taskId = 2a992655-221e-4492-812a-b1d978389a9b task status = finished >2012-11-08 09:36:31,692 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-74) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:36:31,721 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-74) spmStart polling ended, spm status: Free >2012-11-08 09:36:31,737 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-74) START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=2a992655-221e-4492-812a-b1d978389a9b), log id: f082ddf >2012-11-08 09:36:31,757 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-74) FINISH, HSMClearTaskVDSCommand, log id: f082ddf >2012-11-08 09:36:31,757 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-74) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6e730042, log id: 1205e8bb >2012-11-08 09:36:31,759 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-74) [401625b4] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:36:31,762 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-65) [59ff242f] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:36:31,779 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [401625b4] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:36:32,395 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-35) [4c3ca9b3] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:36:33,772 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-79) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:36:34,404 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-69) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:36:35,782 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-90) [5e18cdc1] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:36:36,416 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-86) [78cca392] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:36:37,792 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-91) [5f904976] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:36:38,425 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-81) [2d6e717] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:36:39,801 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-1) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:36:40,435 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-96) [88cae39] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:36:41,823 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-99) [7f760eef] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:36:41,845 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:36:41,874 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:36:41,877 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:36:41,895 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-10) START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 18c298ae >2012-11-08 09:36:41,918 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-10) spmStart polling started: taskId = eb05799a-f651-4727-ba72-dcbc6e5d5b22 >2012-11-08 09:36:42,445 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-17) [36467d8f] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:36:42,953 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-10) Failed in HSMGetTaskStatusVDS method >2012-11-08 09:36:42,953 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-10) Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:36:42,953 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-10) spmStart polling ended: taskId = eb05799a-f651-4727-ba72-dcbc6e5d5b22 task status = finished >2012-11-08 09:36:42,953 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-10) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:36:42,985 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-10) spmStart polling ended, spm status: Free >2012-11-08 09:36:42,999 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-10) START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=eb05799a-f651-4727-ba72-dcbc6e5d5b22), log id: 198cc97a >2012-11-08 09:36:43,017 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-10) FINISH, HSMClearTaskVDSCommand, log id: 198cc97a >2012-11-08 09:36:43,017 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-10) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@69ff75b0, log id: 18c298ae >2012-11-08 09:36:43,019 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-10) [287ac2e] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:36:43,041 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) [287ac2e] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:36:43,835 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-6) [2ed69c19] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:36:44,455 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-2) [3742c089] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:36:45,844 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-11) [25cff2cc] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:36:46,466 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-4) [1856f862] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:36:47,854 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-30) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:36:48,476 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-20) [79d6527d] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:36:49,864 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-31) [6143973] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:36:50,485 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-27) [5064beea] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:36:51,875 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-19) [145ec76c] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:36:52,493 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-39) [6878973a] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:36:53,106 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-29) [745c1fd1] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:36:53,131 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-29) [745c1fd1] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:36:53,135 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-29) [745c1fd1] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:36:53,147 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-29) [745c1fd1] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 945ceee >2012-11-08 09:36:53,169 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-29) [745c1fd1] spmStart polling started: taskId = 0b5b0a43-7cea-440c-918b-a5c819a16e16 >2012-11-08 09:36:53,887 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-37) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:36:54,207 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-29) [745c1fd1] Failed in HSMGetTaskStatusVDS method >2012-11-08 09:36:54,207 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-29) [745c1fd1] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:36:54,207 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-29) [745c1fd1] spmStart polling ended: taskId = 0b5b0a43-7cea-440c-918b-a5c819a16e16 task status = finished >2012-11-08 09:36:54,207 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-29) [745c1fd1] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:36:54,236 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-29) [745c1fd1] spmStart polling ended, spm status: Free >2012-11-08 09:36:54,249 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-29) [745c1fd1] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=0b5b0a43-7cea-440c-918b-a5c819a16e16), log id: 224b6f67 >2012-11-08 09:36:54,271 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-29) [745c1fd1] FINISH, HSMClearTaskVDSCommand, log id: 224b6f67 >2012-11-08 09:36:54,271 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-29) [745c1fd1] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@67ee4c8, log id: 945ceee >2012-11-08 09:36:54,273 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-29) [74de09cc] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:36:54,289 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-29) [74de09cc] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:36:54,504 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-42) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:36:55,898 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-32) [6fe738af] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:36:56,513 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-38) [34b2a1e] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:36:57,922 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-49) [6b4d6902] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:36:58,523 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-53) [24e0f4f0] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:36:59,934 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-55) [367e311c] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:37:00,535 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-58) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:37:01,945 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-68) [77dbb526] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:37:02,544 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-64) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:37:03,954 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-66) [65296545] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:37:04,344 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-93) [51520066] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:37:04,365 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-93) [51520066] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:37:04,368 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-93) [51520066] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:37:04,385 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-93) [51520066] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 23b044e >2012-11-08 09:37:04,409 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-93) [51520066] spmStart polling started: taskId = 63419392-9db9-4f6e-b31f-11fa24e09cf5 >2012-11-08 09:37:04,556 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-76) [2879b44b] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:37:05,439 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-93) [51520066] Failed in HSMGetTaskStatusVDS method >2012-11-08 09:37:05,439 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-93) [51520066] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:37:05,439 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-93) [51520066] spmStart polling ended: taskId = 63419392-9db9-4f6e-b31f-11fa24e09cf5 task status = finished >2012-11-08 09:37:05,439 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-93) [51520066] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:37:05,468 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-93) [51520066] spmStart polling ended, spm status: Free >2012-11-08 09:37:05,486 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-93) [51520066] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=63419392-9db9-4f6e-b31f-11fa24e09cf5), log id: 695b7c87 >2012-11-08 09:37:05,512 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-93) [51520066] FINISH, HSMClearTaskVDSCommand, log id: 695b7c87 >2012-11-08 09:37:05,512 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-93) [51520066] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@466f4a83, log id: 23b044e >2012-11-08 09:37:05,513 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-93) [34850295] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:37:05,532 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-93) [34850295] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:37:05,964 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-65) [59ff242f] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:37:06,567 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-74) [401625b4] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:37:07,974 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-85) [a58ec10] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:37:08,578 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-80) [3dbe1e04] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:37:09,983 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-89) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:37:10,588 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-75) [4599bfd2] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:37:11,994 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-81) [2d6e717] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:37:12,599 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-100) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:37:14,003 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-1) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:37:14,609 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-96) [88cae39] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:37:15,594 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-95) [41e3afd7] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:37:15,623 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-95) [41e3afd7] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:37:15,626 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-95) [41e3afd7] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:37:15,643 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-95) [41e3afd7] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4865861b >2012-11-08 09:37:15,673 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-95) [41e3afd7] spmStart polling started: taskId = f8644880-b850-4db8-8172-79e81ff6775d >2012-11-08 09:37:16,012 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-99) [7f760eef] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:37:16,620 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-17) [36467d8f] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:37:16,713 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-95) [41e3afd7] Failed in HSMGetTaskStatusVDS method >2012-11-08 09:37:16,713 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-95) [41e3afd7] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:37:16,713 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-95) [41e3afd7] spmStart polling ended: taskId = f8644880-b850-4db8-8172-79e81ff6775d task status = finished >2012-11-08 09:37:16,713 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-95) [41e3afd7] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:37:16,745 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-95) [41e3afd7] spmStart polling ended, spm status: Free >2012-11-08 09:37:16,757 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-95) [41e3afd7] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=f8644880-b850-4db8-8172-79e81ff6775d), log id: 34415ee2 >2012-11-08 09:37:16,780 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-95) [41e3afd7] FINISH, HSMClearTaskVDSCommand, log id: 34415ee2 >2012-11-08 09:37:16,780 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-95) [41e3afd7] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2fbc623b, log id: 4865861b >2012-11-08 09:37:16,782 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-95) [78d527f8] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:37:16,796 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-95) [78d527f8] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:37:18,022 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-10) [287ac2e] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:37:18,629 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-8) [6cf7f655] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:37:20,031 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-7) [7361edd9] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:37:20,640 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-15) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:37:22,043 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-20) [79d6527d] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:37:22,648 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-23) [3514bad5] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:37:24,052 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-31) [6143973] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:37:24,658 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-13) [1c2d5b4] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:37:26,063 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-9) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:37:26,667 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-21) [6bd73967] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:37:26,874 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-36) [6a24e609] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:37:26,903 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-36) [6a24e609] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:37:26,906 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-36) [6a24e609] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:37:26,919 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-36) [6a24e609] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 316947b5 >2012-11-08 09:37:26,951 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-36) [6a24e609] spmStart polling started: taskId = d6e65ced-67cf-4873-a620-4392ab30cf6a >2012-11-08 09:37:27,984 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-36) [6a24e609] Failed in HSMGetTaskStatusVDS method >2012-11-08 09:37:27,984 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-36) [6a24e609] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:37:27,984 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-36) [6a24e609] spmStart polling ended: taskId = d6e65ced-67cf-4873-a620-4392ab30cf6a task status = finished >2012-11-08 09:37:27,984 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-36) [6a24e609] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:37:28,011 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-36) [6a24e609] spmStart polling ended, spm status: Free >2012-11-08 09:37:28,026 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-36) [6a24e609] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=d6e65ced-67cf-4873-a620-4392ab30cf6a), log id: 6f6f5ac8 >2012-11-08 09:37:28,044 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-36) [6a24e609] FINISH, HSMClearTaskVDSCommand, log id: 6f6f5ac8 >2012-11-08 09:37:28,044 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-36) [6a24e609] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@77b1399f, log id: 316947b5 >2012-11-08 09:37:28,046 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-36) [143e331d] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:37:28,066 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-36) [143e331d] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:37:28,070 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-37) XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:37:28,678 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-29) [74de09cc] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:37:30,081 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-40) [46f2edae] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:37:30,687 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-38) [34b2a1e] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv15.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:37:32,090 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-49) [6b4d6902] XML RPC error in command GetCapabilitiesVDS ( HostName = rhs-gp-srv12.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, NoRouteToHostException: No route to host >2012-11-08 09:37:33,549 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-53) [4487bbb5] Running command: InitVdsOnUpCommand internal: true. >2012-11-08 09:37:33,561 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-53) [4487bbb5] No string for UNASSIGNED type. Use default Log >2012-11-08 09:37:33,563 INFO [org.ovirt.engine.core.bll.MultipleActionsRunner] (pool-4-thread-50) [4487bbb5] MultipleActionsRunner of type MigrateVmToServer invoked with no actions >2012-11-08 09:37:33,573 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-53) [781c3fe5] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 12e4f376-29af-11e2-b479-52540035ed48 Type: VDS >2012-11-08 09:37:33,582 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-53) [3a904612] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 12e4f376-29af-11e2-b479-52540035ed48 Type: VDS >2012-11-08 09:37:34,983 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-72) [73bac68] Running command: InitVdsOnUpCommand internal: true. >2012-11-08 09:37:34,992 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-72) [73bac68] No string for UNASSIGNED type. Use default Log >2012-11-08 09:37:34,993 INFO [org.ovirt.engine.core.bll.MultipleActionsRunner] (pool-4-thread-50) [73bac68] MultipleActionsRunner of type MigrateVmToServer invoked with no actions >2012-11-08 09:37:34,996 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-72) [6c059950] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 056a9890-29af-11e2-b5a8-52540035ed48 Type: VDS >2012-11-08 09:37:35,000 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-72) [17c5d30e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 056a9890-29af-11e2-b5a8-52540035ed48 Type: VDS >2012-11-08 09:37:38,133 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-64) hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:37:38,162 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-64) SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:37:38,165 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-64) starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:37:38,182 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-64) START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 500d8f7a >2012-11-08 09:37:38,211 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-64) spmStart polling started: taskId = da9ebbf2-9649-41a1-97ea-875d0e710cd8 >2012-11-08 09:37:39,257 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-64) Failed in HSMGetTaskStatusVDS method >2012-11-08 09:37:39,257 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-64) Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:37:39,257 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-64) spmStart polling ended: taskId = da9ebbf2-9649-41a1-97ea-875d0e710cd8 task status = finished >2012-11-08 09:37:39,257 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-64) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:37:39,285 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-64) spmStart polling ended, spm status: Free >2012-11-08 09:37:39,297 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-64) START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=da9ebbf2-9649-41a1-97ea-875d0e710cd8), log id: 32c0d006 >2012-11-08 09:37:39,319 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-64) FINISH, HSMClearTaskVDSCommand, log id: 32c0d006 >2012-11-08 09:37:39,319 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-64) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@109953f2, log id: 500d8f7a >2012-11-08 09:37:39,321 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-64) [42563d61] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:37:39,346 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-64) [42563d61] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:37:49,404 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [88cae39] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:37:49,432 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [88cae39] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:37:49,435 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [88cae39] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:37:49,452 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-96) [88cae39] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 74f989b5 >2012-11-08 09:37:49,481 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-96) [88cae39] spmStart polling started: taskId = 982a02d9-4ed2-4423-9a94-4d1c458022bc >2012-11-08 09:37:50,513 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-96) [88cae39] Failed in HSMGetTaskStatusVDS method >2012-11-08 09:37:50,513 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-96) [88cae39] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:37:50,514 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-96) [88cae39] spmStart polling ended: taskId = 982a02d9-4ed2-4423-9a94-4d1c458022bc task status = finished >2012-11-08 09:37:50,514 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-96) [88cae39] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:37:50,542 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-96) [88cae39] spmStart polling ended, spm status: Free >2012-11-08 09:37:50,560 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-96) [88cae39] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=982a02d9-4ed2-4423-9a94-4d1c458022bc), log id: 53c37cae >2012-11-08 09:37:50,577 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-96) [88cae39] FINISH, HSMClearTaskVDSCommand, log id: 53c37cae >2012-11-08 09:37:50,577 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-96) [88cae39] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@472eae78, log id: 74f989b5 >2012-11-08 09:37:50,580 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-96) [49c06cca] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:37:50,600 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [49c06cca] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:38:00,659 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-19) [145ec76c] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:38:00,684 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-19) [145ec76c] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:38:00,688 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-19) [145ec76c] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:38:00,699 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-19) [145ec76c] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 289aa5f0 >2012-11-08 09:38:00,721 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-19) [145ec76c] spmStart polling started: taskId = 15d45bc2-75d6-4104-90f6-48be76cf18d0 >2012-11-08 09:38:01,755 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-19) [145ec76c] Failed in HSMGetTaskStatusVDS method >2012-11-08 09:38:01,756 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-19) [145ec76c] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:38:01,756 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-19) [145ec76c] spmStart polling ended: taskId = 15d45bc2-75d6-4104-90f6-48be76cf18d0 task status = finished >2012-11-08 09:38:01,756 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-19) [145ec76c] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:38:01,783 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-19) [145ec76c] spmStart polling ended, spm status: Free >2012-11-08 09:38:01,796 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-19) [145ec76c] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=15d45bc2-75d6-4104-90f6-48be76cf18d0), log id: 461f2919 >2012-11-08 09:38:01,815 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-19) [145ec76c] FINISH, HSMClearTaskVDSCommand, log id: 461f2919 >2012-11-08 09:38:01,815 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-19) [145ec76c] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@27c34c46, log id: 289aa5f0 >2012-11-08 09:38:01,816 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-19) [6c49c0f] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:38:01,831 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-19) [6c49c0f] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:38:11,887 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) [4ee90ab8] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:38:11,913 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) [4ee90ab8] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:38:11,916 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) [4ee90ab8] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:38:11,932 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-63) [4ee90ab8] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 32a171e7 >2012-11-08 09:38:11,956 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-63) [4ee90ab8] spmStart polling started: taskId = 1033bb67-2fb5-42db-b55b-d2a6857ca5c1 >2012-11-08 09:38:12,995 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-63) [4ee90ab8] Failed in HSMGetTaskStatusVDS method >2012-11-08 09:38:12,995 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-63) [4ee90ab8] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:38:12,995 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-63) [4ee90ab8] spmStart polling ended: taskId = 1033bb67-2fb5-42db-b55b-d2a6857ca5c1 task status = finished >2012-11-08 09:38:12,995 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-63) [4ee90ab8] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:38:13,024 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-63) [4ee90ab8] spmStart polling ended, spm status: Free >2012-11-08 09:38:13,036 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-63) [4ee90ab8] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=1033bb67-2fb5-42db-b55b-d2a6857ca5c1), log id: 1373b0fa >2012-11-08 09:38:13,058 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-63) [4ee90ab8] FINISH, HSMClearTaskVDSCommand, log id: 1373b0fa >2012-11-08 09:38:13,058 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-63) [4ee90ab8] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1067f710, log id: 32a171e7 >2012-11-08 09:38:13,060 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-63) [198d43da] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:38:13,076 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-63) [198d43da] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:38:16,106 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (ajp-/127.0.0.1:8702-11) [5bd7c912] START, ValidateStorageServerConnectionVDSCommand(HostName = rhs-gp-srv12.lab.eng.blr.redhat.com, HostId = 056a9890-29af-11e2-b5a8-52540035ed48, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = POSIXFS, connectionList = [{ id: null, connection: rhs-client1.lab.eng.blr.redhat.com:/dis-rep, iqn: null, vfsType: glusterfs, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 74bd12ca >2012-11-08 09:38:16,129 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (ajp-/127.0.0.1:8702-11) [5bd7c912] FINISH, ValidateStorageServerConnectionVDSCommand, return: {00000000-0000-0000-0000-000000000000=0}, log id: 74bd12ca >2012-11-08 09:38:16,134 INFO [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-11) [5bd7c912] Running command: AddStorageServerConnectionCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2012-11-08 09:38:16,136 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-11) [5bd7c912] START, ConnectStorageServerVDSCommand(HostName = rhs-gp-srv12.lab.eng.blr.redhat.com, HostId = 056a9890-29af-11e2-b5a8-52540035ed48, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = POSIXFS, connectionList = [{ id: null, connection: rhs-client1.lab.eng.blr.redhat.com:/dis-rep, iqn: null, vfsType: glusterfs, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 75c7abe6 >2012-11-08 09:38:16,227 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-11) [5bd7c912] FINISH, ConnectStorageServerVDSCommand, return: {00000000-0000-0000-0000-000000000000=100}, log id: 75c7abe6 >2012-11-08 09:38:16,229 ERROR [org.ovirt.engine.core.bll.storage.POSIXFSStorageHelper] (ajp-/127.0.0.1:8702-11) [5bd7c912] The connection with details rhs-client1.lab.eng.blr.redhat.com:/dis-rep failed because of error code 100 and error message is: general exception >2012-11-08 09:38:16,255 WARN [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (ajp-/127.0.0.1:8702-10) [34614102] The message key AddPosixFsStorageDomain is missing from bundles/ExecutionMessages >2012-11-08 09:38:16,265 WARN [org.ovirt.engine.core.bll.storage.AddPosixFsStorageDomainCommand] (ajp-/127.0.0.1:8702-10) [34614102] CanDoAction of action AddPosixFsStorageDomain failed. Reasons:VAR__ACTION__ADD,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_STORAGE_CONNECTION_NOT_EXIST >2012-11-08 09:38:16,327 INFO [org.ovirt.engine.core.bll.storage.RemoveStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-6) [3c07ce0] Running command: RemoveStorageServerConnectionCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2012-11-08 09:38:16,328 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-6) [3c07ce0] START, DisconnectStorageServerVDSCommand(HostName = rhs-gp-srv12.lab.eng.blr.redhat.com, HostId = 056a9890-29af-11e2-b5a8-52540035ed48, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = POSIXFS, connectionList = [{ id: null, connection: rhs-client1.lab.eng.blr.redhat.com:/dis-rep, iqn: null, vfsType: glusterfs, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 10949e40 >2012-11-08 09:38:18,631 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-6) [3c07ce0] FINISH, DisconnectStorageServerVDSCommand, return: {00000000-0000-0000-0000-000000000000=477}, log id: 10949e40 >2012-11-08 09:38:23,137 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [642523c8] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:38:23,162 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [642523c8] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:38:23,165 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [642523c8] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:38:23,181 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-84) [642523c8] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 21a16125 >2012-11-08 09:38:23,210 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-84) [642523c8] spmStart polling started: taskId = 445d61e4-1ea9-4507-a40c-a333a068e800 >2012-11-08 09:38:24,250 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-84) [642523c8] Failed in HSMGetTaskStatusVDS method >2012-11-08 09:38:24,250 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-84) [642523c8] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:38:24,250 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-84) [642523c8] spmStart polling ended: taskId = 445d61e4-1ea9-4507-a40c-a333a068e800 task status = finished >2012-11-08 09:38:24,250 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-84) [642523c8] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:38:24,279 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-84) [642523c8] spmStart polling ended, spm status: Free >2012-11-08 09:38:24,293 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-84) [642523c8] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=445d61e4-1ea9-4507-a40c-a333a068e800), log id: 4a71132b >2012-11-08 09:38:24,311 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-84) [642523c8] FINISH, HSMClearTaskVDSCommand, log id: 4a71132b >2012-11-08 09:38:24,311 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-84) [642523c8] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@45a1923b, log id: 21a16125 >2012-11-08 09:38:24,313 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-84) [72878333] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:38:24,342 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [72878333] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:38:34,401 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-9) hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:38:34,424 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-9) SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:38:34,427 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-9) starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:38:34,442 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-9) START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 536801d9 >2012-11-08 09:38:34,470 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-9) spmStart polling started: taskId = feee7975-0c76-45a6-a148-a9de44a9d2ee >2012-11-08 09:38:35,506 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-9) Failed in HSMGetTaskStatusVDS method >2012-11-08 09:38:35,506 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-9) Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:38:35,506 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-9) spmStart polling ended: taskId = feee7975-0c76-45a6-a148-a9de44a9d2ee task status = finished >2012-11-08 09:38:35,506 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-9) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:38:35,533 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-9) spmStart polling ended, spm status: Free >2012-11-08 09:38:35,545 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-9) START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=feee7975-0c76-45a6-a148-a9de44a9d2ee), log id: 451a5338 >2012-11-08 09:38:35,567 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-9) FINISH, HSMClearTaskVDSCommand, log id: 451a5338 >2012-11-08 09:38:35,567 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-9) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@434da4f9, log id: 536801d9 >2012-11-08 09:38:35,568 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-9) [1e89f9f4] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:38:35,595 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-9) [1e89f9f4] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:38:45,656 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:38:45,682 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:38:45,707 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:38:45,719 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 21f70010 >2012-11-08 09:38:45,748 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) spmStart polling started: taskId = c4cd22be-9983-4de2-9cb7-d8301a4454f2 >2012-11-08 09:38:46,788 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-46) Failed in HSMGetTaskStatusVDS method >2012-11-08 09:38:46,788 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-46) Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:38:46,788 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) spmStart polling ended: taskId = c4cd22be-9983-4de2-9cb7-d8301a4454f2 task status = finished >2012-11-08 09:38:46,788 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:38:46,813 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) spmStart polling ended, spm status: Free >2012-11-08 09:38:46,825 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-46) START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=c4cd22be-9983-4de2-9cb7-d8301a4454f2), log id: 27ffa24c >2012-11-08 09:38:46,847 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-46) FINISH, HSMClearTaskVDSCommand, log id: 27ffa24c >2012-11-08 09:38:46,847 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7ff84017, log id: 21f70010 >2012-11-08 09:38:46,849 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-46) [337e7a4] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:38:46,875 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [337e7a4] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:38:56,923 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [7ad09500] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:38:56,950 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [7ad09500] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:38:56,952 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [7ad09500] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:38:56,969 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-97) [7ad09500] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5b222cc2 >2012-11-08 09:38:56,998 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-97) [7ad09500] spmStart polling started: taskId = 3c930af8-cb70-46e5-82cf-31570a5792b4 >2012-11-08 09:38:58,036 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-97) [7ad09500] Failed in HSMGetTaskStatusVDS method >2012-11-08 09:38:58,036 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-97) [7ad09500] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:38:58,036 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-97) [7ad09500] spmStart polling ended: taskId = 3c930af8-cb70-46e5-82cf-31570a5792b4 task status = finished >2012-11-08 09:38:58,036 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-97) [7ad09500] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:38:58,067 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-97) [7ad09500] spmStart polling ended, spm status: Free >2012-11-08 09:38:58,081 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-97) [7ad09500] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=3c930af8-cb70-46e5-82cf-31570a5792b4), log id: c4f3ba2 >2012-11-08 09:38:58,106 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-97) [7ad09500] FINISH, HSMClearTaskVDSCommand, log id: c4f3ba2 >2012-11-08 09:38:58,106 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-97) [7ad09500] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1a7f529c, log id: 5b222cc2 >2012-11-08 09:38:58,108 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-97) [1d088d5b] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:38:58,132 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [1d088d5b] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:39:08,200 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-22) [5fe2db58] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:39:08,228 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-22) [5fe2db58] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:39:08,231 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-22) [5fe2db58] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:39:08,248 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-22) [5fe2db58] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7f10d09e >2012-11-08 09:39:08,278 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-22) [5fe2db58] spmStart polling started: taskId = f2db5dfe-b260-4780-837a-10dcd68076b8 >2012-11-08 09:39:09,315 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-22) [5fe2db58] Failed in HSMGetTaskStatusVDS method >2012-11-08 09:39:09,315 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-22) [5fe2db58] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:39:09,316 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-22) [5fe2db58] spmStart polling ended: taskId = f2db5dfe-b260-4780-837a-10dcd68076b8 task status = finished >2012-11-08 09:39:09,316 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-22) [5fe2db58] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:39:09,342 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-22) [5fe2db58] spmStart polling ended, spm status: Free >2012-11-08 09:39:09,354 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-22) [5fe2db58] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=f2db5dfe-b260-4780-837a-10dcd68076b8), log id: 77f248a7 >2012-11-08 09:39:09,375 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-22) [5fe2db58] FINISH, HSMClearTaskVDSCommand, log id: 77f248a7 >2012-11-08 09:39:09,375 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-22) [5fe2db58] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@43800976, log id: 7f10d09e >2012-11-08 09:39:09,377 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-22) [aed4499] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:39:09,396 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-22) [aed4499] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:39:14,511 INFO [org.ovirt.engine.core.bll.SetVmTicketCommand] (ajp-/127.0.0.1:8702-2) [6582a82b] Running command: SetVmTicketCommand internal: false. Entities affected : ID: 67bd2d56-8e0b-4328-bcfd-ea7f4758c885 Type: VM >2012-11-08 09:39:14,532 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp-/127.0.0.1:8702-2) [6582a82b] START, SetVmTicketVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=67bd2d56-8e0b-4328-bcfd-ea7f4758c885, ticket=fC5uB4mJzCuG, validTime=120,m userName=admin@internal, userId=fdfc627c-d875-11e0-90f0-83df133b58cc), log id: 2f8ba39b >2012-11-08 09:39:14,574 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp-/127.0.0.1:8702-2) [6582a82b] FINISH, SetVmTicketVDSCommand, log id: 2f8ba39b >2012-11-08 09:39:15,821 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (ajp-/127.0.0.1:8702-3) [455990a1] START, ValidateStorageServerConnectionVDSCommand(HostName = rhs-gp-srv12.lab.eng.blr.redhat.com, HostId = 056a9890-29af-11e2-b5a8-52540035ed48, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = POSIXFS, connectionList = [{ id: null, connection: rhs-client1.lab.eng.blr.redhat.com:/dis-rep, iqn: null, vfsType: glusterfs, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 3482543 >2012-11-08 09:39:15,845 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (ajp-/127.0.0.1:8702-3) [455990a1] FINISH, ValidateStorageServerConnectionVDSCommand, return: {00000000-0000-0000-0000-000000000000=0}, log id: 3482543 >2012-11-08 09:39:15,849 INFO [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-3) [455990a1] Running command: AddStorageServerConnectionCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2012-11-08 09:39:15,851 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-3) [455990a1] START, ConnectStorageServerVDSCommand(HostName = rhs-gp-srv12.lab.eng.blr.redhat.com, HostId = 056a9890-29af-11e2-b5a8-52540035ed48, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = POSIXFS, connectionList = [{ id: null, connection: rhs-client1.lab.eng.blr.redhat.com:/dis-rep, iqn: null, vfsType: glusterfs, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 7e87cc49 >2012-11-08 09:39:15,903 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-3) [455990a1] FINISH, ConnectStorageServerVDSCommand, return: {00000000-0000-0000-0000-000000000000=100}, log id: 7e87cc49 >2012-11-08 09:39:15,905 ERROR [org.ovirt.engine.core.bll.storage.POSIXFSStorageHelper] (ajp-/127.0.0.1:8702-3) [455990a1] The connection with details rhs-client1.lab.eng.blr.redhat.com:/dis-rep failed because of error code 100 and error message is: general exception >2012-11-08 09:39:15,934 WARN [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (ajp-/127.0.0.1:8702-1) [6ff833b7] The message key AddPosixFsStorageDomain is missing from bundles/ExecutionMessages >2012-11-08 09:39:15,947 WARN [org.ovirt.engine.core.bll.storage.AddPosixFsStorageDomainCommand] (ajp-/127.0.0.1:8702-1) [6ff833b7] CanDoAction of action AddPosixFsStorageDomain failed. Reasons:VAR__ACTION__ADD,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_STORAGE_CONNECTION_NOT_EXIST >2012-11-08 09:39:16,025 INFO [org.ovirt.engine.core.bll.storage.RemoveStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-14) [3ec869f2] Running command: RemoveStorageServerConnectionCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2012-11-08 09:39:16,027 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-14) [3ec869f2] START, DisconnectStorageServerVDSCommand(HostName = rhs-gp-srv12.lab.eng.blr.redhat.com, HostId = 056a9890-29af-11e2-b5a8-52540035ed48, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = POSIXFS, connectionList = [{ id: null, connection: rhs-client1.lab.eng.blr.redhat.com:/dis-rep, iqn: null, vfsType: glusterfs, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 487aebcf >2012-11-08 09:39:18,297 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-14) [3ec869f2] FINISH, DisconnectStorageServerVDSCommand, return: {00000000-0000-0000-0000-000000000000=477}, log id: 487aebcf >2012-11-08 09:39:19,444 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-58) hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:39:19,470 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-58) SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:39:19,473 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-58) starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:39:19,485 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-58) START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: bd8d452 >2012-11-08 09:39:19,506 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-58) spmStart polling started: taskId = 0f48436e-527b-4ecd-ac37-cdf573e313c3 >2012-11-08 09:39:20,546 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-58) Failed in HSMGetTaskStatusVDS method >2012-11-08 09:39:20,547 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-58) Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:39:20,547 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-58) spmStart polling ended: taskId = 0f48436e-527b-4ecd-ac37-cdf573e313c3 task status = finished >2012-11-08 09:39:20,547 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-58) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:39:20,575 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-58) spmStart polling ended, spm status: Free >2012-11-08 09:39:20,590 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-58) START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=0f48436e-527b-4ecd-ac37-cdf573e313c3), log id: ddbbf5d >2012-11-08 09:39:20,607 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-58) FINISH, HSMClearTaskVDSCommand, log id: ddbbf5d >2012-11-08 09:39:20,608 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-58) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@55f079c6, log id: bd8d452 >2012-11-08 09:39:20,609 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-58) [1ecae23e] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:39:20,625 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-58) [1ecae23e] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:39:30,680 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-91) [5f904976] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:39:30,702 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-91) [5f904976] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:39:30,705 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-91) [5f904976] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:39:30,720 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-91) [5f904976] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 59af955 >2012-11-08 09:39:30,741 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-91) [5f904976] spmStart polling started: taskId = 206a7376-5d3f-4390-a62e-188723893db5 >2012-11-08 09:39:31,403 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (ajp-/127.0.0.1:8702-10) [4e4c3d10] START, ValidateStorageServerConnectionVDSCommand(HostName = rhs-gp-srv12.lab.eng.blr.redhat.com, HostId = 056a9890-29af-11e2-b5a8-52540035ed48, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = POSIXFS, connectionList = [{ id: null, connection: rhs-client1.lab.eng.blr.redhat.com:/replicate, iqn: null, vfsType: glusterfs, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 65446d >2012-11-08 09:39:31,422 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (ajp-/127.0.0.1:8702-10) [4e4c3d10] FINISH, ValidateStorageServerConnectionVDSCommand, return: {00000000-0000-0000-0000-000000000000=0}, log id: 65446d >2012-11-08 09:39:31,427 INFO [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-10) [4e4c3d10] Running command: AddStorageServerConnectionCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2012-11-08 09:39:31,429 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-10) [4e4c3d10] START, ConnectStorageServerVDSCommand(HostName = rhs-gp-srv12.lab.eng.blr.redhat.com, HostId = 056a9890-29af-11e2-b5a8-52540035ed48, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = POSIXFS, connectionList = [{ id: null, connection: rhs-client1.lab.eng.blr.redhat.com:/replicate, iqn: null, vfsType: glusterfs, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 39ed46a4 >2012-11-08 09:39:31,466 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-10) [4e4c3d10] FINISH, ConnectStorageServerVDSCommand, return: {00000000-0000-0000-0000-000000000000=100}, log id: 39ed46a4 >2012-11-08 09:39:31,468 ERROR [org.ovirt.engine.core.bll.storage.POSIXFSStorageHelper] (ajp-/127.0.0.1:8702-10) [4e4c3d10] The connection with details rhs-client1.lab.eng.blr.redhat.com:/replicate failed because of error code 100 and error message is: general exception >2012-11-08 09:39:31,492 WARN [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (ajp-/127.0.0.1:8702-9) [36b06b45] The message key AddPosixFsStorageDomain is missing from bundles/ExecutionMessages >2012-11-08 09:39:31,505 WARN [org.ovirt.engine.core.bll.storage.AddPosixFsStorageDomainCommand] (ajp-/127.0.0.1:8702-9) [36b06b45] CanDoAction of action AddPosixFsStorageDomain failed. Reasons:VAR__ACTION__ADD,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_STORAGE_CONNECTION_NOT_EXIST >2012-11-08 09:39:31,605 INFO [org.ovirt.engine.core.bll.storage.RemoveStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-1) [272b3063] Running command: RemoveStorageServerConnectionCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2012-11-08 09:39:31,606 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-1) [272b3063] START, DisconnectStorageServerVDSCommand(HostName = rhs-gp-srv12.lab.eng.blr.redhat.com, HostId = 056a9890-29af-11e2-b5a8-52540035ed48, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = POSIXFS, connectionList = [{ id: null, connection: rhs-client1.lab.eng.blr.redhat.com:/replicate, iqn: null, vfsType: glusterfs, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 43adf7c5 >2012-11-08 09:39:31,771 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-91) [5f904976] Failed in HSMGetTaskStatusVDS method >2012-11-08 09:39:31,771 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-91) [5f904976] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:39:31,771 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-91) [5f904976] spmStart polling ended: taskId = 206a7376-5d3f-4390-a62e-188723893db5 task status = finished >2012-11-08 09:39:31,771 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-91) [5f904976] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:39:31,799 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-91) [5f904976] spmStart polling ended, spm status: Free >2012-11-08 09:39:31,811 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-91) [5f904976] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=206a7376-5d3f-4390-a62e-188723893db5), log id: 2f68d499 >2012-11-08 09:39:31,829 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-91) [5f904976] FINISH, HSMClearTaskVDSCommand, log id: 2f68d499 >2012-11-08 09:39:31,829 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-91) [5f904976] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@b5da120, log id: 59af955 >2012-11-08 09:39:31,830 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-91) [2ffd41b4] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:39:31,851 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-91) [2ffd41b4] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:39:33,869 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-1) [272b3063] FINISH, DisconnectStorageServerVDSCommand, return: {00000000-0000-0000-0000-000000000000=477}, log id: 43adf7c5 >2012-11-08 09:39:41,901 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) [6143973] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:39:41,928 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) [6143973] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:39:41,931 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) [6143973] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:39:41,947 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-31) [6143973] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5b33448b >2012-11-08 09:39:41,975 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-31) [6143973] spmStart polling started: taskId = f0f9a980-a860-43d6-8018-faad5459104d >2012-11-08 09:39:43,017 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-31) [6143973] Failed in HSMGetTaskStatusVDS method >2012-11-08 09:39:43,017 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-31) [6143973] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:39:43,017 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-31) [6143973] spmStart polling ended: taskId = f0f9a980-a860-43d6-8018-faad5459104d task status = finished >2012-11-08 09:39:43,017 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-31) [6143973] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:39:43,045 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-31) [6143973] spmStart polling ended, spm status: Free >2012-11-08 09:39:43,069 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-31) [6143973] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=f0f9a980-a860-43d6-8018-faad5459104d), log id: 5895c0ee >2012-11-08 09:39:43,087 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-31) [6143973] FINISH, HSMClearTaskVDSCommand, log id: 5895c0ee >2012-11-08 09:39:43,087 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-31) [6143973] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1c834022, log id: 5b33448b >2012-11-08 09:39:43,090 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-31) [6ce786cd] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:39:43,113 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) [6ce786cd] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:39:53,173 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [17c5d30e] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:39:53,193 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [17c5d30e] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:39:53,196 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [17c5d30e] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:39:53,215 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [17c5d30e] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6cc0d47c >2012-11-08 09:39:53,240 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [17c5d30e] spmStart polling started: taskId = d11eac06-8fb6-4e51-b0a6-6c7fc4e12190 >2012-11-08 09:39:54,277 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-72) [17c5d30e] Failed in HSMGetTaskStatusVDS method >2012-11-08 09:39:54,277 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-72) [17c5d30e] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:39:54,277 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [17c5d30e] spmStart polling ended: taskId = d11eac06-8fb6-4e51-b0a6-6c7fc4e12190 task status = finished >2012-11-08 09:39:54,277 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [17c5d30e] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:39:54,305 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [17c5d30e] spmStart polling ended, spm status: Free >2012-11-08 09:39:54,319 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-72) [17c5d30e] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=d11eac06-8fb6-4e51-b0a6-6c7fc4e12190), log id: 10e9c02 >2012-11-08 09:39:54,342 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-72) [17c5d30e] FINISH, HSMClearTaskVDSCommand, log id: 10e9c02 >2012-11-08 09:39:54,342 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [17c5d30e] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@662447d8, log id: 6cc0d47c >2012-11-08 09:39:54,344 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-72) [57e8ea4b] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:39:54,361 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [57e8ea4b] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:39:59,705 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (ajp-/127.0.0.1:8702-8) [5782fc32] START, ValidateStorageServerConnectionVDSCommand(HostName = rhs-gp-srv12.lab.eng.blr.redhat.com, HostId = 056a9890-29af-11e2-b5a8-52540035ed48, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = POSIXFS, connectionList = [{ id: null, connection: rhs-client16.lab.eng.blr.redhat.com:/replicate, iqn: null, vfsType: glusterfs, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 6f86d92d >2012-11-08 09:39:59,732 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (ajp-/127.0.0.1:8702-8) [5782fc32] FINISH, ValidateStorageServerConnectionVDSCommand, return: {00000000-0000-0000-0000-000000000000=0}, log id: 6f86d92d >2012-11-08 09:39:59,736 INFO [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-8) [5782fc32] Running command: AddStorageServerConnectionCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2012-11-08 09:39:59,739 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-8) [5782fc32] START, ConnectStorageServerVDSCommand(HostName = rhs-gp-srv12.lab.eng.blr.redhat.com, HostId = 056a9890-29af-11e2-b5a8-52540035ed48, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = POSIXFS, connectionList = [{ id: null, connection: rhs-client16.lab.eng.blr.redhat.com:/replicate, iqn: null, vfsType: glusterfs, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: c0739ce >2012-11-08 09:39:59,780 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-8) [5782fc32] FINISH, ConnectStorageServerVDSCommand, return: {00000000-0000-0000-0000-000000000000=100}, log id: c0739ce >2012-11-08 09:39:59,781 ERROR [org.ovirt.engine.core.bll.storage.POSIXFSStorageHelper] (ajp-/127.0.0.1:8702-8) [5782fc32] The connection with details rhs-client16.lab.eng.blr.redhat.com:/replicate failed because of error code 100 and error message is: general exception >2012-11-08 09:39:59,804 WARN [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (ajp-/127.0.0.1:8702-14) [580c04f4] The message key AddPosixFsStorageDomain is missing from bundles/ExecutionMessages >2012-11-08 09:39:59,817 WARN [org.ovirt.engine.core.bll.storage.AddPosixFsStorageDomainCommand] (ajp-/127.0.0.1:8702-14) [580c04f4] CanDoAction of action AddPosixFsStorageDomain failed. Reasons:VAR__ACTION__ADD,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_STORAGE_CONNECTION_NOT_EXIST >2012-11-08 09:39:59,918 INFO [org.ovirt.engine.core.bll.storage.RemoveStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-2) [28d621b5] Running command: RemoveStorageServerConnectionCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2012-11-08 09:39:59,920 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-2) [28d621b5] START, DisconnectStorageServerVDSCommand(HostName = rhs-gp-srv12.lab.eng.blr.redhat.com, HostId = 056a9890-29af-11e2-b5a8-52540035ed48, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = POSIXFS, connectionList = [{ id: null, connection: rhs-client16.lab.eng.blr.redhat.com:/replicate, iqn: null, vfsType: glusterfs, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 5aea6de3 >2012-11-08 09:40:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-79) Autorecovering hosts is disabled, skipping >2012-11-08 09:40:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-79) Autorecovering storage domains is disabled, skipping >2012-11-08 09:40:02,197 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-2) [28d621b5] FINISH, DisconnectStorageServerVDSCommand, return: {00000000-0000-0000-0000-000000000000=477}, log id: 5aea6de3 >2012-11-08 09:40:04,423 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:40:04,448 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:40:04,452 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:40:04,467 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-1) START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 45bf7987 >2012-11-08 09:40:04,490 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-1) spmStart polling started: taskId = eeb23918-77b3-479f-b565-e822635f99a2 >2012-11-08 09:40:05,520 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-1) Failed in HSMGetTaskStatusVDS method >2012-11-08 09:40:05,520 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-1) Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:40:05,520 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-1) spmStart polling ended: taskId = eeb23918-77b3-479f-b565-e822635f99a2 task status = finished >2012-11-08 09:40:05,520 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-1) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:40:05,546 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-1) spmStart polling ended, spm status: Free >2012-11-08 09:40:05,564 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-1) START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=eeb23918-77b3-479f-b565-e822635f99a2), log id: 55f80f35 >2012-11-08 09:40:05,581 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-1) FINISH, HSMClearTaskVDSCommand, log id: 55f80f35 >2012-11-08 09:40:05,581 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-1) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@10ae8cab, log id: 45bf7987 >2012-11-08 09:40:05,583 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-1) [3395371b] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:40:05,608 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [3395371b] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:40:15,666 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [5064beea] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:40:15,694 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [5064beea] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:40:15,697 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [5064beea] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:40:15,709 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-27) [5064beea] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 30f63816 >2012-11-08 09:40:15,732 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-27) [5064beea] spmStart polling started: taskId = 5cd28a78-a926-4e11-b235-f9aec1531118 >2012-11-08 09:40:16,766 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-27) [5064beea] Failed in HSMGetTaskStatusVDS method >2012-11-08 09:40:16,766 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-27) [5064beea] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:40:16,766 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-27) [5064beea] spmStart polling ended: taskId = 5cd28a78-a926-4e11-b235-f9aec1531118 task status = finished >2012-11-08 09:40:16,766 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-27) [5064beea] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:40:16,795 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-27) [5064beea] spmStart polling ended, spm status: Free >2012-11-08 09:40:16,808 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-27) [5064beea] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=5cd28a78-a926-4e11-b235-f9aec1531118), log id: 23c425f9 >2012-11-08 09:40:16,826 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-27) [5064beea] FINISH, HSMClearTaskVDSCommand, log id: 23c425f9 >2012-11-08 09:40:16,826 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-27) [5064beea] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@229910c7, log id: 30f63816 >2012-11-08 09:40:16,828 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-27) [6f1c8a8f] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:40:16,850 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [6f1c8a8f] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:40:26,896 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [3f1726c1] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:40:26,924 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [3f1726c1] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:40:26,927 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [3f1726c1] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:40:26,944 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) [3f1726c1] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 28d294f4 >2012-11-08 09:40:26,966 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) [3f1726c1] spmStart polling started: taskId = 54bafa6f-7393-4f8c-aa41-05fbd8c7639d >2012-11-08 09:40:27,998 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-61) [3f1726c1] Failed in HSMGetTaskStatusVDS method >2012-11-08 09:40:27,998 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-61) [3f1726c1] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:40:27,999 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) [3f1726c1] spmStart polling ended: taskId = 54bafa6f-7393-4f8c-aa41-05fbd8c7639d task status = finished >2012-11-08 09:40:27,999 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) [3f1726c1] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:40:28,026 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) [3f1726c1] spmStart polling ended, spm status: Free >2012-11-08 09:40:28,039 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-61) [3f1726c1] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=54bafa6f-7393-4f8c-aa41-05fbd8c7639d), log id: 37f38fdf >2012-11-08 09:40:28,057 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-61) [3f1726c1] FINISH, HSMClearTaskVDSCommand, log id: 37f38fdf >2012-11-08 09:40:28,057 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) [3f1726c1] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@576a962b, log id: 28d294f4 >2012-11-08 09:40:28,059 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-61) [16a90de3] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:40:28,084 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [16a90de3] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:40:38,168 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [3089800c] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:40:38,209 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [3089800c] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:40:38,212 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [3089800c] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:40:38,231 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-87) [3089800c] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4d57e8b6 >2012-11-08 09:40:38,278 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-87) [3089800c] spmStart polling started: taskId = dd822415-bc75-41fb-ac0a-bb53f1ad5c5c >2012-11-08 09:40:39,319 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-87) [3089800c] Failed in HSMGetTaskStatusVDS method >2012-11-08 09:40:39,320 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-87) [3089800c] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:40:39,320 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-87) [3089800c] spmStart polling ended: taskId = dd822415-bc75-41fb-ac0a-bb53f1ad5c5c task status = finished >2012-11-08 09:40:39,320 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-87) [3089800c] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:40:39,345 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-87) [3089800c] spmStart polling ended, spm status: Free >2012-11-08 09:40:39,358 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-87) [3089800c] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=dd822415-bc75-41fb-ac0a-bb53f1ad5c5c), log id: 5ebb9177 >2012-11-08 09:40:39,381 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-87) [3089800c] FINISH, HSMClearTaskVDSCommand, log id: 5ebb9177 >2012-11-08 09:40:39,381 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-87) [3089800c] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6c4b4c37, log id: 4d57e8b6 >2012-11-08 09:40:39,382 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-87) [4378e66c] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:40:39,399 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [4378e66c] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:40:49,452 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [23551d18] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:40:49,482 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [23551d18] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:40:49,485 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [23551d18] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:40:49,504 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [23551d18] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 1d5e92df >2012-11-08 09:40:49,526 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [23551d18] spmStart polling started: taskId = 2ebcdaca-da88-4573-97b7-5caef731fa6f >2012-11-08 09:40:50,582 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-25) [23551d18] Failed in HSMGetTaskStatusVDS method >2012-11-08 09:40:50,582 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-25) [23551d18] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:40:50,583 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [23551d18] spmStart polling ended: taskId = 2ebcdaca-da88-4573-97b7-5caef731fa6f task status = finished >2012-11-08 09:40:50,583 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [23551d18] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:40:50,628 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [23551d18] spmStart polling ended, spm status: Free >2012-11-08 09:40:50,640 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-25) [23551d18] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=2ebcdaca-da88-4573-97b7-5caef731fa6f), log id: 51bfd8d3 >2012-11-08 09:40:50,682 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-25) [23551d18] FINISH, HSMClearTaskVDSCommand, log id: 51bfd8d3 >2012-11-08 09:40:50,682 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [23551d18] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1721e4b7, log id: 1d5e92df >2012-11-08 09:40:50,684 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-25) [322eb63e] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:40:50,703 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [322eb63e] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:41:00,753 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [709f6bac] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:41:00,782 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [709f6bac] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:41:00,786 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [709f6bac] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:41:00,803 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-57) [709f6bac] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5e708a50 >2012-11-08 09:41:00,824 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-57) [709f6bac] spmStart polling started: taskId = 86d67a82-a6bb-4965-a678-e9021ef20d9c >2012-11-08 09:41:01,856 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-57) [709f6bac] Failed in HSMGetTaskStatusVDS method >2012-11-08 09:41:01,856 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-57) [709f6bac] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:41:01,857 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-57) [709f6bac] spmStart polling ended: taskId = 86d67a82-a6bb-4965-a678-e9021ef20d9c task status = finished >2012-11-08 09:41:01,857 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-57) [709f6bac] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:41:01,884 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-57) [709f6bac] spmStart polling ended, spm status: Free >2012-11-08 09:41:01,902 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-57) [709f6bac] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=86d67a82-a6bb-4965-a678-e9021ef20d9c), log id: 1225fb25 >2012-11-08 09:41:01,919 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-57) [709f6bac] FINISH, HSMClearTaskVDSCommand, log id: 1225fb25 >2012-11-08 09:41:01,919 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-57) [709f6bac] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7b8b4371, log id: 5e708a50 >2012-11-08 09:41:01,921 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-57) [73f8ef14] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:41:01,944 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [73f8ef14] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:41:12,001 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-88) [5fa94a95] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:41:12,030 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-88) [5fa94a95] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:41:12,033 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-88) [5fa94a95] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:41:12,052 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-88) [5fa94a95] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 273fe895 >2012-11-08 09:41:12,081 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-88) [5fa94a95] spmStart polling started: taskId = a7ff0d82-0d63-4e55-b57c-2c3355282c88 >2012-11-08 09:41:13,113 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-88) [5fa94a95] Failed in HSMGetTaskStatusVDS method >2012-11-08 09:41:13,114 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-88) [5fa94a95] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:41:13,114 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-88) [5fa94a95] spmStart polling ended: taskId = a7ff0d82-0d63-4e55-b57c-2c3355282c88 task status = finished >2012-11-08 09:41:13,114 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-88) [5fa94a95] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:41:13,143 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-88) [5fa94a95] spmStart polling ended, spm status: Free >2012-11-08 09:41:13,161 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-88) [5fa94a95] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=a7ff0d82-0d63-4e55-b57c-2c3355282c88), log id: 97cb715 >2012-11-08 09:41:13,187 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-88) [5fa94a95] FINISH, HSMClearTaskVDSCommand, log id: 97cb715 >2012-11-08 09:41:13,187 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-88) [5fa94a95] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@5ad7c890, log id: 273fe895 >2012-11-08 09:41:13,189 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-88) [cb09488] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:41:13,215 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-88) [cb09488] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:41:23,288 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [3514bad5] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:41:23,333 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [3514bad5] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:41:23,337 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [3514bad5] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:41:23,353 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [3514bad5] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 72806f6e >2012-11-08 09:41:23,397 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [3514bad5] spmStart polling started: taskId = b42c49c3-0b11-4719-82b3-f77218399656 >2012-11-08 09:41:24,428 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-23) [3514bad5] Failed in HSMGetTaskStatusVDS method >2012-11-08 09:41:24,428 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-23) [3514bad5] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:41:24,428 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [3514bad5] spmStart polling ended: taskId = b42c49c3-0b11-4719-82b3-f77218399656 task status = finished >2012-11-08 09:41:24,428 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [3514bad5] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:41:24,454 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [3514bad5] spmStart polling ended, spm status: Free >2012-11-08 09:41:24,471 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-23) [3514bad5] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=b42c49c3-0b11-4719-82b3-f77218399656), log id: 27fb28cc >2012-11-08 09:41:24,490 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-23) [3514bad5] FINISH, HSMClearTaskVDSCommand, log id: 27fb28cc >2012-11-08 09:41:24,490 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [3514bad5] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@5a337d95, log id: 72806f6e >2012-11-08 09:41:24,493 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-23) [6a79ef76] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:41:24,512 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [6a79ef76] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:41:34,581 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:41:34,605 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:41:34,616 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:41:34,632 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-59) START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 518effa9 >2012-11-08 09:41:34,660 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-59) spmStart polling started: taskId = af0a838d-5298-439c-8d93-b227cb425ff2 >2012-11-08 09:41:35,698 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-59) Failed in HSMGetTaskStatusVDS method >2012-11-08 09:41:35,698 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-59) Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:41:35,698 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-59) spmStart polling ended: taskId = af0a838d-5298-439c-8d93-b227cb425ff2 task status = finished >2012-11-08 09:41:35,698 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-59) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:41:35,726 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-59) spmStart polling ended, spm status: Free >2012-11-08 09:41:35,742 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-59) START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=af0a838d-5298-439c-8d93-b227cb425ff2), log id: 1b331627 >2012-11-08 09:41:35,767 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-59) FINISH, HSMClearTaskVDSCommand, log id: 1b331627 >2012-11-08 09:41:35,767 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-59) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@36f36864, log id: 518effa9 >2012-11-08 09:41:35,769 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-59) [264a996b] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:41:35,794 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [264a996b] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:41:45,852 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) [78cca392] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:41:45,880 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) [78cca392] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:41:45,883 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) [78cca392] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:41:45,901 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-86) [78cca392] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 378c7b6 >2012-11-08 09:41:45,926 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-86) [78cca392] spmStart polling started: taskId = 85c0ee26-e7a5-467e-a0fa-515947c6f24a >2012-11-08 09:41:46,956 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-86) [78cca392] Failed in HSMGetTaskStatusVDS method >2012-11-08 09:41:46,956 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-86) [78cca392] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:41:46,956 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-86) [78cca392] spmStart polling ended: taskId = 85c0ee26-e7a5-467e-a0fa-515947c6f24a task status = finished >2012-11-08 09:41:46,956 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-86) [78cca392] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:41:46,987 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-86) [78cca392] spmStart polling ended, spm status: Free >2012-11-08 09:41:47,002 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-86) [78cca392] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=85c0ee26-e7a5-467e-a0fa-515947c6f24a), log id: 59cebc84 >2012-11-08 09:41:47,021 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-86) [78cca392] FINISH, HSMClearTaskVDSCommand, log id: 59cebc84 >2012-11-08 09:41:47,021 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-86) [78cca392] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@13baac2c, log id: 378c7b6 >2012-11-08 09:41:47,023 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-86) [447d86d2] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:41:47,045 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) [447d86d2] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:41:57,110 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-28) hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:41:57,139 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-28) SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:41:57,142 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-28) starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:41:57,160 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-28) START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3e448d43 >2012-11-08 09:41:57,190 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-28) spmStart polling started: taskId = 27fc892a-7bbf-46b7-bc40-e1b2c1654134 >2012-11-08 09:41:58,222 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-28) Failed in HSMGetTaskStatusVDS method >2012-11-08 09:41:58,222 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-28) Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:41:58,222 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-28) spmStart polling ended: taskId = 27fc892a-7bbf-46b7-bc40-e1b2c1654134 task status = finished >2012-11-08 09:41:58,222 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-28) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:41:58,251 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-28) spmStart polling ended, spm status: Free >2012-11-08 09:41:58,264 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-28) START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=27fc892a-7bbf-46b7-bc40-e1b2c1654134), log id: 496758db >2012-11-08 09:41:58,287 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-28) FINISH, HSMClearTaskVDSCommand, log id: 496758db >2012-11-08 09:41:58,287 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-28) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1ffab783, log id: 3e448d43 >2012-11-08 09:41:58,289 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-28) [52609dc] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:41:58,307 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-28) [52609dc] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:42:08,374 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-43) [698db2f6] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:42:08,396 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-43) [698db2f6] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:42:08,399 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-43) [698db2f6] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:42:08,418 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-43) [698db2f6] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 29d05f3f >2012-11-08 09:42:08,449 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-43) [698db2f6] spmStart polling started: taskId = 4da23806-c9fb-4651-a8c7-b28d1f99de02 >2012-11-08 09:42:09,487 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-43) [698db2f6] Failed in HSMGetTaskStatusVDS method >2012-11-08 09:42:09,487 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-43) [698db2f6] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:42:09,487 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-43) [698db2f6] spmStart polling ended: taskId = 4da23806-c9fb-4651-a8c7-b28d1f99de02 task status = finished >2012-11-08 09:42:09,487 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-43) [698db2f6] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:42:09,520 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-43) [698db2f6] spmStart polling ended, spm status: Free >2012-11-08 09:42:09,533 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-43) [698db2f6] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=4da23806-c9fb-4651-a8c7-b28d1f99de02), log id: 52aa11fa >2012-11-08 09:42:09,551 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-43) [698db2f6] FINISH, HSMClearTaskVDSCommand, log id: 52aa11fa >2012-11-08 09:42:09,551 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-43) [698db2f6] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1b825e7c, log id: 29d05f3f >2012-11-08 09:42:09,553 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-43) [c092d00] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:42:09,575 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-43) [c092d00] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:42:19,642 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:42:19,670 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:42:19,673 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:42:19,690 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-98) START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6cb3ea14 >2012-11-08 09:42:19,720 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-98) spmStart polling started: taskId = fc7938ab-db60-47f7-bf97-66197292ba75 >2012-11-08 09:42:20,764 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-98) Failed in HSMGetTaskStatusVDS method >2012-11-08 09:42:20,764 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-98) Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:42:20,765 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-98) spmStart polling ended: taskId = fc7938ab-db60-47f7-bf97-66197292ba75 task status = finished >2012-11-08 09:42:20,765 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-98) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:42:20,793 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-98) spmStart polling ended, spm status: Free >2012-11-08 09:42:20,805 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-98) START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=fc7938ab-db60-47f7-bf97-66197292ba75), log id: 4d880b9c >2012-11-08 09:42:20,827 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-98) FINISH, HSMClearTaskVDSCommand, log id: 4d880b9c >2012-11-08 09:42:20,827 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-98) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1eb25688, log id: 6cb3ea14 >2012-11-08 09:42:20,829 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-98) [4e4f0a76] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:42:20,853 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [4e4f0a76] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:42:30,909 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [7361edd9] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:42:30,938 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [7361edd9] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:42:30,941 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [7361edd9] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:42:30,960 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [7361edd9] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: c256883 >2012-11-08 09:42:30,989 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [7361edd9] spmStart polling started: taskId = ec588507-72d2-4496-bd6b-dc1edd714ce4 >2012-11-08 09:42:32,026 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-7) [7361edd9] Failed in HSMGetTaskStatusVDS method >2012-11-08 09:42:32,026 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-7) [7361edd9] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:42:32,026 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [7361edd9] spmStart polling ended: taskId = ec588507-72d2-4496-bd6b-dc1edd714ce4 task status = finished >2012-11-08 09:42:32,026 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [7361edd9] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:42:32,057 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [7361edd9] spmStart polling ended, spm status: Free >2012-11-08 09:42:32,071 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-7) [7361edd9] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=ec588507-72d2-4496-bd6b-dc1edd714ce4), log id: 366d17ce >2012-11-08 09:42:32,090 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-7) [7361edd9] FINISH, HSMClearTaskVDSCommand, log id: 366d17ce >2012-11-08 09:42:32,090 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [7361edd9] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3435f14c, log id: c256883 >2012-11-08 09:42:32,092 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-7) [4174cf5b] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:42:32,114 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [4174cf5b] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:42:42,167 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-49) [6b4d6902] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:42:42,198 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-49) [6b4d6902] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:42:42,201 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-49) [6b4d6902] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:42:42,218 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-49) [6b4d6902] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6b47c823 >2012-11-08 09:42:42,240 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-49) [6b4d6902] spmStart polling started: taskId = ea5f1ab8-aa39-423c-91af-a9bfa77f0a17 >2012-11-08 09:42:43,271 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-49) [6b4d6902] Failed in HSMGetTaskStatusVDS method >2012-11-08 09:42:43,271 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-49) [6b4d6902] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:42:43,272 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-49) [6b4d6902] spmStart polling ended: taskId = ea5f1ab8-aa39-423c-91af-a9bfa77f0a17 task status = finished >2012-11-08 09:42:43,272 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-49) [6b4d6902] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:42:43,298 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-49) [6b4d6902] spmStart polling ended, spm status: Free >2012-11-08 09:42:43,312 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-49) [6b4d6902] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=ea5f1ab8-aa39-423c-91af-a9bfa77f0a17), log id: 130d9dcd >2012-11-08 09:42:43,330 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-49) [6b4d6902] FINISH, HSMClearTaskVDSCommand, log id: 130d9dcd >2012-11-08 09:42:43,330 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-49) [6b4d6902] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@62f30780, log id: 6b47c823 >2012-11-08 09:42:43,331 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-49) [41719740] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:42:43,350 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-49) [41719740] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:42:53,417 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:42:53,446 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:42:53,449 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:42:53,467 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 241d810b >2012-11-08 09:42:53,496 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) spmStart polling started: taskId = 31c5df8b-3e6c-4049-9e1c-78cac1f4fa8d >2012-11-08 09:42:54,542 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-69) Failed in HSMGetTaskStatusVDS method >2012-11-08 09:42:54,542 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-69) Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:42:54,542 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) spmStart polling ended: taskId = 31c5df8b-3e6c-4049-9e1c-78cac1f4fa8d task status = finished >2012-11-08 09:42:54,542 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:42:54,569 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) spmStart polling ended, spm status: Free >2012-11-08 09:42:54,582 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-69) START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=31c5df8b-3e6c-4049-9e1c-78cac1f4fa8d), log id: 41ff9c0f >2012-11-08 09:42:54,605 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-69) FINISH, HSMClearTaskVDSCommand, log id: 41ff9c0f >2012-11-08 09:42:54,605 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@745ebea2, log id: 241d810b >2012-11-08 09:42:54,607 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-69) [7835b95d] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:42:54,632 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [7835b95d] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:43:04,686 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) [3a2a7aa9] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:43:04,715 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) [3a2a7aa9] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:43:04,718 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) [3a2a7aa9] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:43:04,731 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-51) [3a2a7aa9] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3bed71a2 >2012-11-08 09:43:04,752 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-51) [3a2a7aa9] spmStart polling started: taskId = 440592f7-9760-4f98-9af9-9ba2746845e2 >2012-11-08 09:43:05,781 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-51) [3a2a7aa9] Failed in HSMGetTaskStatusVDS method >2012-11-08 09:43:05,781 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-51) [3a2a7aa9] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:43:05,781 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-51) [3a2a7aa9] spmStart polling ended: taskId = 440592f7-9760-4f98-9af9-9ba2746845e2 task status = finished >2012-11-08 09:43:05,781 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-51) [3a2a7aa9] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:43:05,808 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-51) [3a2a7aa9] spmStart polling ended, spm status: Free >2012-11-08 09:43:05,821 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-51) [3a2a7aa9] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=440592f7-9760-4f98-9af9-9ba2746845e2), log id: 2a7bee50 >2012-11-08 09:43:05,839 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-51) [3a2a7aa9] FINISH, HSMClearTaskVDSCommand, log id: 2a7bee50 >2012-11-08 09:43:05,839 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-51) [3a2a7aa9] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2e5adbfa, log id: 3bed71a2 >2012-11-08 09:43:05,841 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-51) [4f2011dc] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:43:05,861 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) [4f2011dc] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:43:15,918 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:43:15,947 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:43:15,950 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:43:15,967 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5b177940 >2012-11-08 09:43:15,988 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) spmStart polling started: taskId = f0d02e38-3a1c-4094-a6a8-c3bbb02b8021 >2012-11-08 09:43:17,019 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-94) Failed in HSMGetTaskStatusVDS method >2012-11-08 09:43:17,019 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-94) Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:43:17,019 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) spmStart polling ended: taskId = f0d02e38-3a1c-4094-a6a8-c3bbb02b8021 task status = finished >2012-11-08 09:43:17,019 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:43:17,048 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) spmStart polling ended, spm status: Free >2012-11-08 09:43:17,066 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-94) START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=f0d02e38-3a1c-4094-a6a8-c3bbb02b8021), log id: 68676555 >2012-11-08 09:43:17,083 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-94) FINISH, HSMClearTaskVDSCommand, log id: 68676555 >2012-11-08 09:43:17,083 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@5c2de4ff, log id: 5b177940 >2012-11-08 09:43:17,085 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-94) [491550a0] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:43:17,109 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [491550a0] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:43:27,156 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-79) hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:43:27,185 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-79) SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:43:27,188 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-79) starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:43:27,202 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-79) START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5124e985 >2012-11-08 09:43:27,225 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-79) spmStart polling started: taskId = 916b498b-f3f5-4893-84d9-d15a2e1dda5b >2012-11-08 09:43:28,259 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-79) Failed in HSMGetTaskStatusVDS method >2012-11-08 09:43:28,259 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-79) Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:43:28,259 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-79) spmStart polling ended: taskId = 916b498b-f3f5-4893-84d9-d15a2e1dda5b task status = finished >2012-11-08 09:43:28,259 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-79) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:43:28,288 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-79) spmStart polling ended, spm status: Free >2012-11-08 09:43:28,305 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-79) START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=916b498b-f3f5-4893-84d9-d15a2e1dda5b), log id: 2d54351f >2012-11-08 09:43:28,322 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-79) FINISH, HSMClearTaskVDSCommand, log id: 2d54351f >2012-11-08 09:43:28,322 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-79) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@736b9d78, log id: 5124e985 >2012-11-08 09:43:28,325 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-79) [1e2f6c7b] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:43:28,348 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-79) [1e2f6c7b] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:43:38,408 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [6cf7f655] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:43:38,436 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [6cf7f655] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:43:38,439 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [6cf7f655] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:43:38,456 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [6cf7f655] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7520dd55 >2012-11-08 09:43:38,480 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [6cf7f655] spmStart polling started: taskId = 612db819-42e8-4aa6-b809-bd6d7a0a383b >2012-11-08 09:43:39,515 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-8) [6cf7f655] Failed in HSMGetTaskStatusVDS method >2012-11-08 09:43:39,515 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-8) [6cf7f655] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:43:39,515 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [6cf7f655] spmStart polling ended: taskId = 612db819-42e8-4aa6-b809-bd6d7a0a383b task status = finished >2012-11-08 09:43:39,515 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [6cf7f655] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:43:39,554 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [6cf7f655] spmStart polling ended, spm status: Free >2012-11-08 09:43:39,572 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-8) [6cf7f655] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=612db819-42e8-4aa6-b809-bd6d7a0a383b), log id: 7951e2fa >2012-11-08 09:43:39,601 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-8) [6cf7f655] FINISH, HSMClearTaskVDSCommand, log id: 7951e2fa >2012-11-08 09:43:39,601 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [6cf7f655] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@499546de, log id: 7520dd55 >2012-11-08 09:43:39,604 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-8) [3ed7d34e] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:43:39,627 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [3ed7d34e] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:43:49,682 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-32) [6fe738af] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:43:49,712 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-32) [6fe738af] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:43:49,716 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-32) [6fe738af] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:43:49,734 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-32) [6fe738af] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2bc3b242 >2012-11-08 09:43:49,768 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-32) [6fe738af] spmStart polling started: taskId = 111c5e61-8b13-42e1-99bd-47f07f8a2fc4 >2012-11-08 09:43:50,814 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-32) [6fe738af] Failed in HSMGetTaskStatusVDS method >2012-11-08 09:43:50,814 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-32) [6fe738af] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:43:50,814 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-32) [6fe738af] spmStart polling ended: taskId = 111c5e61-8b13-42e1-99bd-47f07f8a2fc4 task status = finished >2012-11-08 09:43:50,814 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-32) [6fe738af] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:43:50,844 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-32) [6fe738af] spmStart polling ended, spm status: Free >2012-11-08 09:43:50,858 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-32) [6fe738af] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=111c5e61-8b13-42e1-99bd-47f07f8a2fc4), log id: c448de0 >2012-11-08 09:43:50,876 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-32) [6fe738af] FINISH, HSMClearTaskVDSCommand, log id: c448de0 >2012-11-08 09:43:50,877 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-32) [6fe738af] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@68be736a, log id: 2bc3b242 >2012-11-08 09:43:50,878 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-32) [2f8b69df] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:43:50,897 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-32) [2f8b69df] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:44:00,957 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [a58ec10] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:44:00,980 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [a58ec10] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:44:00,983 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [a58ec10] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:44:00,997 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [a58ec10] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 64d74969 >2012-11-08 09:44:01,020 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [a58ec10] spmStart polling started: taskId = 36d748b2-6e12-47c5-a406-c90bf222c4aa >2012-11-08 09:44:02,052 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-85) [a58ec10] Failed in HSMGetTaskStatusVDS method >2012-11-08 09:44:02,052 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-85) [a58ec10] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:44:02,052 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [a58ec10] spmStart polling ended: taskId = 36d748b2-6e12-47c5-a406-c90bf222c4aa task status = finished >2012-11-08 09:44:02,052 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [a58ec10] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:44:02,079 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [a58ec10] spmStart polling ended, spm status: Free >2012-11-08 09:44:02,090 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-85) [a58ec10] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=36d748b2-6e12-47c5-a406-c90bf222c4aa), log id: eda193c >2012-11-08 09:44:02,112 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-85) [a58ec10] FINISH, HSMClearTaskVDSCommand, log id: eda193c >2012-11-08 09:44:02,112 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [a58ec10] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@10aeb9c2, log id: 64d74969 >2012-11-08 09:44:02,114 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-85) [3f25781c] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:44:02,141 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [3f25781c] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:44:12,190 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-18) [7ab5f38b] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:44:12,220 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-18) [7ab5f38b] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:44:12,222 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-18) [7ab5f38b] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:44:12,241 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-18) [7ab5f38b] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 71131a36 >2012-11-08 09:44:12,272 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-18) [7ab5f38b] spmStart polling started: taskId = 4a6b6055-0127-40e3-84c6-291041774a7e >2012-11-08 09:44:13,306 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-18) [7ab5f38b] Failed in HSMGetTaskStatusVDS method >2012-11-08 09:44:13,306 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-18) [7ab5f38b] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:44:13,306 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-18) [7ab5f38b] spmStart polling ended: taskId = 4a6b6055-0127-40e3-84c6-291041774a7e task status = finished >2012-11-08 09:44:13,306 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-18) [7ab5f38b] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:44:13,334 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-18) [7ab5f38b] spmStart polling ended, spm status: Free >2012-11-08 09:44:13,347 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-18) [7ab5f38b] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=4a6b6055-0127-40e3-84c6-291041774a7e), log id: 2d344583 >2012-11-08 09:44:13,366 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-18) [7ab5f38b] FINISH, HSMClearTaskVDSCommand, log id: 2d344583 >2012-11-08 09:44:13,366 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-18) [7ab5f38b] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@42bfb2a4, log id: 71131a36 >2012-11-08 09:44:13,368 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-18) [37694f38] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:44:13,392 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-18) [37694f38] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:44:23,454 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-33) hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:44:23,483 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-33) SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:44:23,486 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-33) starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:44:23,503 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-33) START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 33ea1707 >2012-11-08 09:44:23,534 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-33) spmStart polling started: taskId = 6996fb3a-75ce-468d-b82d-5dcbfa7236e6 >2012-11-08 09:44:24,565 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-33) Failed in HSMGetTaskStatusVDS method >2012-11-08 09:44:24,565 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-33) Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:44:24,565 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-33) spmStart polling ended: taskId = 6996fb3a-75ce-468d-b82d-5dcbfa7236e6 task status = finished >2012-11-08 09:44:24,565 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-33) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:44:24,594 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-33) spmStart polling ended, spm status: Free >2012-11-08 09:44:24,607 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-33) START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=6996fb3a-75ce-468d-b82d-5dcbfa7236e6), log id: 418fa802 >2012-11-08 09:44:24,630 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-33) FINISH, HSMClearTaskVDSCommand, log id: 418fa802 >2012-11-08 09:44:24,630 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-33) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3e5f3cee, log id: 33ea1707 >2012-11-08 09:44:24,632 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-33) [5e994d3f] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:44:24,650 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-33) [5e994d3f] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:44:34,704 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) [59ff242f] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:44:34,732 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) [59ff242f] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:44:34,735 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) [59ff242f] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:44:34,753 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-65) [59ff242f] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2512f83c >2012-11-08 09:44:34,774 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-65) [59ff242f] spmStart polling started: taskId = ab8ebccf-5161-40b1-879d-fc55e4447bce >2012-11-08 09:44:35,805 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-65) [59ff242f] Failed in HSMGetTaskStatusVDS method >2012-11-08 09:44:35,805 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-65) [59ff242f] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:44:35,805 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-65) [59ff242f] spmStart polling ended: taskId = ab8ebccf-5161-40b1-879d-fc55e4447bce task status = finished >2012-11-08 09:44:35,805 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-65) [59ff242f] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:44:35,833 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-65) [59ff242f] spmStart polling ended, spm status: Free >2012-11-08 09:44:35,843 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-65) [59ff242f] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=ab8ebccf-5161-40b1-879d-fc55e4447bce), log id: 15a8009e >2012-11-08 09:44:35,861 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-65) [59ff242f] FINISH, HSMClearTaskVDSCommand, log id: 15a8009e >2012-11-08 09:44:35,861 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-65) [59ff242f] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@48c0902d, log id: 2512f83c >2012-11-08 09:44:35,864 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-65) [5a399fab] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:44:35,885 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) [5a399fab] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:44:45,943 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [36467d8f] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:44:45,972 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [36467d8f] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:44:45,976 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [36467d8f] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:44:45,993 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-17) [36467d8f] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 49a87ced >2012-11-08 09:44:46,017 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-17) [36467d8f] spmStart polling started: taskId = 656c0d6a-df9d-47fd-a649-e0c4d02584a7 >2012-11-08 09:44:47,055 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-17) [36467d8f] Failed in HSMGetTaskStatusVDS method >2012-11-08 09:44:47,055 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-17) [36467d8f] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:44:47,055 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-17) [36467d8f] spmStart polling ended: taskId = 656c0d6a-df9d-47fd-a649-e0c4d02584a7 task status = finished >2012-11-08 09:44:47,055 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-17) [36467d8f] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:44:47,084 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-17) [36467d8f] spmStart polling ended, spm status: Free >2012-11-08 09:44:47,096 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-17) [36467d8f] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=656c0d6a-df9d-47fd-a649-e0c4d02584a7), log id: 6b9b2b54 >2012-11-08 09:44:47,119 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-17) [36467d8f] FINISH, HSMClearTaskVDSCommand, log id: 6b9b2b54 >2012-11-08 09:44:47,119 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-17) [36467d8f] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1839cefd, log id: 49a87ced >2012-11-08 09:44:47,121 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-17) [7b445077] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:44:47,138 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [7b445077] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:44:57,195 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-44) [64124e53] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:44:57,222 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-44) [64124e53] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:44:57,225 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-44) [64124e53] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:44:57,237 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-44) [64124e53] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2d15126b >2012-11-08 09:44:57,259 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-44) [64124e53] spmStart polling started: taskId = be264f61-e29f-4d05-b92a-bf24cd7941f0 >2012-11-08 09:44:58,289 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-44) [64124e53] Failed in HSMGetTaskStatusVDS method >2012-11-08 09:44:58,289 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-44) [64124e53] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:44:58,289 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-44) [64124e53] spmStart polling ended: taskId = be264f61-e29f-4d05-b92a-bf24cd7941f0 task status = finished >2012-11-08 09:44:58,289 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-44) [64124e53] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:44:58,315 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-44) [64124e53] spmStart polling ended, spm status: Free >2012-11-08 09:44:58,332 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-44) [64124e53] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=be264f61-e29f-4d05-b92a-bf24cd7941f0), log id: 303cfa20 >2012-11-08 09:44:58,348 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-44) [64124e53] FINISH, HSMClearTaskVDSCommand, log id: 303cfa20 >2012-11-08 09:44:58,348 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-44) [64124e53] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7349639d, log id: 2d15126b >2012-11-08 09:44:58,351 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-44) [572b138c] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:44:58,375 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-44) [572b138c] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:45:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-52) [63bd4906] Autorecovering hosts is disabled, skipping >2012-11-08 09:45:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-52) [63bd4906] Autorecovering storage domains is disabled, skipping >2012-11-08 09:45:08,433 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:45:08,456 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:45:08,458 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:45:08,473 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-82) START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6ffd8e0c >2012-11-08 09:45:08,501 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-82) spmStart polling started: taskId = 51995ff7-2ed5-4223-95b8-12709edd5895 >2012-11-08 09:45:09,538 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-82) Failed in HSMGetTaskStatusVDS method >2012-11-08 09:45:09,539 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-82) Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:45:09,539 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-82) spmStart polling ended: taskId = 51995ff7-2ed5-4223-95b8-12709edd5895 task status = finished >2012-11-08 09:45:09,541 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-82) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:45:09,571 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-82) spmStart polling ended, spm status: Free >2012-11-08 09:45:09,590 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-82) START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=51995ff7-2ed5-4223-95b8-12709edd5895), log id: ad32902 >2012-11-08 09:45:09,606 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-82) FINISH, HSMClearTaskVDSCommand, log id: ad32902 >2012-11-08 09:45:09,606 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-82) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2d2947db, log id: 6ffd8e0c >2012-11-08 09:45:09,609 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-82) [712c9206] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:45:09,627 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [712c9206] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:45:19,686 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) [287ac2e] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:45:19,713 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) [287ac2e] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:45:19,715 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) [287ac2e] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:45:19,732 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-10) [287ac2e] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 424d6108 >2012-11-08 09:45:19,761 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-10) [287ac2e] spmStart polling started: taskId = 91fe201d-41ae-4332-82e9-d8a127eabe5f >2012-11-08 09:45:20,792 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-10) [287ac2e] Failed in HSMGetTaskStatusVDS method >2012-11-08 09:45:20,793 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-10) [287ac2e] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:45:20,793 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-10) [287ac2e] spmStart polling ended: taskId = 91fe201d-41ae-4332-82e9-d8a127eabe5f task status = finished >2012-11-08 09:45:20,793 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-10) [287ac2e] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:45:20,816 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-10) [287ac2e] spmStart polling ended, spm status: Free >2012-11-08 09:45:20,827 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-10) [287ac2e] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=91fe201d-41ae-4332-82e9-d8a127eabe5f), log id: 265c0a13 >2012-11-08 09:45:20,849 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-10) [287ac2e] FINISH, HSMClearTaskVDSCommand, log id: 265c0a13 >2012-11-08 09:45:20,849 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-10) [287ac2e] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@64997911, log id: 424d6108 >2012-11-08 09:45:20,850 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-10) [17db7eb0] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:45:20,877 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) [17db7eb0] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:45:22,197 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (ajp-/127.0.0.1:8702-2) [25171114] START, ValidateStorageServerConnectionVDSCommand(HostName = rhs-gp-srv12.lab.eng.blr.redhat.com, HostId = 056a9890-29af-11e2-b5a8-52540035ed48, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = POSIXFS, connectionList = [{ id: null, connection: rhs-client16.lab.eng.blr.redhat.com:/replicate, iqn: null, vfsType: glusterfs, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 548b4faf >2012-11-08 09:45:22,223 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (ajp-/127.0.0.1:8702-2) [25171114] FINISH, ValidateStorageServerConnectionVDSCommand, return: {00000000-0000-0000-0000-000000000000=0}, log id: 548b4faf >2012-11-08 09:45:22,227 INFO [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-2) [25171114] Running command: AddStorageServerConnectionCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2012-11-08 09:45:22,230 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-2) [25171114] START, ConnectStorageServerVDSCommand(HostName = rhs-gp-srv12.lab.eng.blr.redhat.com, HostId = 056a9890-29af-11e2-b5a8-52540035ed48, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = POSIXFS, connectionList = [{ id: null, connection: rhs-client16.lab.eng.blr.redhat.com:/replicate, iqn: null, vfsType: glusterfs, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 608d7844 >2012-11-08 09:45:26,433 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-2) [25171114] FINISH, ConnectStorageServerVDSCommand, return: {00000000-0000-0000-0000-000000000000=0}, log id: 608d7844 >2012-11-08 09:45:26,463 WARN [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (ajp-/127.0.0.1:8702-3) [30ba4c51] The message key AddPosixFsStorageDomain is missing from bundles/ExecutionMessages >2012-11-08 09:45:26,481 INFO [org.ovirt.engine.core.bll.storage.AddPosixFsStorageDomainCommand] (ajp-/127.0.0.1:8702-3) [30ba4c51] Running command: AddPosixFsStorageDomainCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2012-11-08 09:45:26,487 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-3) [30ba4c51] START, CreateStorageDomainVDSCommand(HostName = rhs-gp-srv12.lab.eng.blr.redhat.com, HostId = 056a9890-29af-11e2-b5a8-52540035ed48, storageDomain=org.ovirt.engine.core.common.businessentities.storage_domain_static@c782c9c1, args=rhs-client16.lab.eng.blr.redhat.com:/replicate), log id: 34c455a0 >2012-11-08 09:45:30,274 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-3) [30ba4c51] FINISH, CreateStorageDomainVDSCommand, log id: 34c455a0 >2012-11-08 09:45:30,276 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStorageDomainStatsVDSCommand] (ajp-/127.0.0.1:8702-3) [30ba4c51] START, GetStorageDomainStatsVDSCommand(HostName = rhs-gp-srv12.lab.eng.blr.redhat.com, HostId = 056a9890-29af-11e2-b5a8-52540035ed48, storageDomainId=2769f0af-c1ca-4cb1-8470-d8cc2e830f7c), log id: 661ed852 >2012-11-08 09:45:30,303 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStorageDomainStatsVDSCommand] (ajp-/127.0.0.1:8702-3) [30ba4c51] FINISH, GetStorageDomainStatsVDSCommand, return: org.ovirt.engine.core.common.businessentities.storage_domains@609d9781, log id: 661ed852 >2012-11-08 09:45:30,417 INFO [org.ovirt.engine.core.bll.storage.AttachStorageDomainToPoolCommand] (ajp-/127.0.0.1:8702-12) [7019e5cb] Running command: AttachStorageDomainToPoolCommand internal: false. Entities affected : ID: 2769f0af-c1ca-4cb1-8470-d8cc2e830f7c Type: Storage >2012-11-08 09:45:30,419 INFO [org.ovirt.engine.core.bll.storage.AddStoragePoolWithStoragesCommand] (ajp-/127.0.0.1:8702-12) [2feaccda] Lock Acquired to object EngineLock [exclusiveLocks= key: 1eaedbbb-1c0e-4028-9b0e-adac7e07d25f value: POOL >, sharedLocks= ] >2012-11-08 09:45:30,433 INFO [org.ovirt.engine.core.bll.storage.AddStoragePoolWithStoragesCommand] (ajp-/127.0.0.1:8702-12) [2feaccda] Running command: AddStoragePoolWithStoragesCommand internal: true. Entities affected : ID: 1eaedbbb-1c0e-4028-9b0e-adac7e07d25f Type: StoragePool >2012-11-08 09:45:30,475 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (ajp-/127.0.0.1:8702-12) [246af8df] START, ValidateStorageServerConnectionVDSCommand(HostName = rhs-gp-srv12.lab.eng.blr.redhat.com, HostId = 056a9890-29af-11e2-b5a8-52540035ed48, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = POSIXFS, connectionList = [{ id: 67153a10-6d5c-4c55-a678-075721832deb, connection: rhs-client16.lab.eng.blr.redhat.com:/replicate, iqn: null, vfsType: glusterfs, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 43698d7a >2012-11-08 09:45:30,494 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (ajp-/127.0.0.1:8702-12) [246af8df] FINISH, ValidateStorageServerConnectionVDSCommand, return: {67153a10-6d5c-4c55-a678-075721832deb=0}, log id: 43698d7a >2012-11-08 09:45:30,495 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (ajp-/127.0.0.1:8702-12) [246af8df] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2012-11-08 09:45:30,496 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-12) [246af8df] START, ConnectStorageServerVDSCommand(HostName = rhs-gp-srv12.lab.eng.blr.redhat.com, HostId = 056a9890-29af-11e2-b5a8-52540035ed48, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = POSIXFS, connectionList = [{ id: 67153a10-6d5c-4c55-a678-075721832deb, connection: rhs-client16.lab.eng.blr.redhat.com:/replicate, iqn: null, vfsType: glusterfs, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 1016eaec >2012-11-08 09:45:30,529 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-12) [246af8df] FINISH, ConnectStorageServerVDSCommand, return: {67153a10-6d5c-4c55-a678-075721832deb=0}, log id: 1016eaec >2012-11-08 09:45:30,532 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStoragePoolVDSCommand] (ajp-/127.0.0.1:8702-12) [246af8df] START, CreateStoragePoolVDSCommand(HostName = rhs-gp-srv12.lab.eng.blr.redhat.com, HostId = 056a9890-29af-11e2-b5a8-52540035ed48, storagePoolId=1eaedbbb-1c0e-4028-9b0e-adac7e07d25f, storageType=POSIXFS, storagePoolName=RHS-SELF-HEAL, masterDomainId=2769f0af-c1ca-4cb1-8470-d8cc2e830f7c, domainsIdList=[2769f0af-c1ca-4cb1-8470-d8cc2e830f7c], masterVersion=1), log id: 103060f1 >2012-11-08 09:45:30,948 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-29) [74de09cc] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:45:30,970 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-29) [74de09cc] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:45:30,974 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-29) [74de09cc] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:45:30,991 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-29) [74de09cc] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 29021aa >2012-11-08 09:45:31,020 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-29) [74de09cc] spmStart polling started: taskId = ecefe872-4dba-46ec-a8df-173372e816b1 >2012-11-08 09:45:32,058 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-29) [74de09cc] Failed in HSMGetTaskStatusVDS method >2012-11-08 09:45:32,058 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-29) [74de09cc] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:45:32,058 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-29) [74de09cc] spmStart polling ended: taskId = ecefe872-4dba-46ec-a8df-173372e816b1 task status = finished >2012-11-08 09:45:32,058 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-29) [74de09cc] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:45:32,089 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-29) [74de09cc] spmStart polling ended, spm status: Free >2012-11-08 09:45:32,103 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-29) [74de09cc] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=ecefe872-4dba-46ec-a8df-173372e816b1), log id: 7cdfa046 >2012-11-08 09:45:32,127 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-29) [74de09cc] FINISH, HSMClearTaskVDSCommand, log id: 7cdfa046 >2012-11-08 09:45:32,127 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-29) [74de09cc] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3d9393b6, log id: 29021aa >2012-11-08 09:45:32,129 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-29) [23c6b14f] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:45:32,154 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-29) [23c6b14f] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:45:42,213 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:45:42,234 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:45:42,237 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:45:42,252 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 1a07a52a >2012-11-08 09:45:42,273 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) spmStart polling started: taskId = adc0bafe-feb7-4e02-af0b-c16eb1020340 >2012-11-08 09:45:43,304 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-83) Failed in HSMGetTaskStatusVDS method >2012-11-08 09:45:43,304 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-83) Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:45:43,304 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) spmStart polling ended: taskId = adc0bafe-feb7-4e02-af0b-c16eb1020340 task status = finished >2012-11-08 09:45:43,304 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:45:43,331 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) spmStart polling ended, spm status: Free >2012-11-08 09:45:43,349 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-83) START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=adc0bafe-feb7-4e02-af0b-c16eb1020340), log id: 3f50aff1 >2012-11-08 09:45:43,366 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-83) FINISH, HSMClearTaskVDSCommand, log id: 3f50aff1 >2012-11-08 09:45:43,366 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4a4f17b0, log id: 1a07a52a >2012-11-08 09:45:43,369 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-83) [e9c49ed] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:45:43,394 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [e9c49ed] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:45:53,454 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [89ac041] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:45:53,480 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [89ac041] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:45:53,483 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [89ac041] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:45:53,495 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [89ac041] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6f2a419e >2012-11-08 09:45:53,521 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [89ac041] spmStart polling started: taskId = 492c4bf9-79ad-4912-b19d-cfcd0bfb3f79 >2012-11-08 09:45:54,563 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-14) [89ac041] Failed in HSMGetTaskStatusVDS method >2012-11-08 09:45:54,564 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-14) [89ac041] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:45:54,564 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [89ac041] spmStart polling ended: taskId = 492c4bf9-79ad-4912-b19d-cfcd0bfb3f79 task status = finished >2012-11-08 09:45:54,564 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [89ac041] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:45:54,590 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [89ac041] spmStart polling ended, spm status: Free >2012-11-08 09:45:54,603 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-14) [89ac041] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=492c4bf9-79ad-4912-b19d-cfcd0bfb3f79), log id: 64aa80eb >2012-11-08 09:45:54,620 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-14) [89ac041] FINISH, HSMClearTaskVDSCommand, log id: 64aa80eb >2012-11-08 09:45:54,621 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [89ac041] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@75f5f7ee, log id: 6f2a419e >2012-11-08 09:45:54,622 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-14) [676866c2] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:45:54,645 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [676866c2] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:45:54,848 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStoragePoolVDSCommand] (ajp-/127.0.0.1:8702-12) [246af8df] FINISH, CreateStoragePoolVDSCommand, log id: 103060f1 >2012-11-08 09:45:54,850 INFO [org.ovirt.engine.core.bll.storage.AddStoragePoolWithStoragesCommand] (ajp-/127.0.0.1:8702-12) [246af8df] Lock freed to object EngineLock [exclusiveLocks= key: 1eaedbbb-1c0e-4028-9b0e-adac7e07d25f value: POOL >, sharedLocks= ] >2012-11-08 09:45:54,851 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp-/127.0.0.1:8702-12) [617a9872] Lock Acquired to object EngineLock [exclusiveLocks= key: 2769f0af-c1ca-4cb1-8470-d8cc2e830f7c value: STORAGE >, sharedLocks= ] >2012-11-08 09:45:54,885 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp-/127.0.0.1:8702-12) [617a9872] Running command: ActivateStorageDomainCommand internal: true. Entities affected : ID: 2769f0af-c1ca-4cb1-8470-d8cc2e830f7c Type: Storage >2012-11-08 09:45:54,891 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp-/127.0.0.1:8702-12) [617a9872] Lock freed to object EngineLock [exclusiveLocks= key: 2769f0af-c1ca-4cb1-8470-d8cc2e830f7c value: STORAGE >, sharedLocks= ] >2012-11-08 09:45:54,891 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp-/127.0.0.1:8702-12) [617a9872] ActivateStorage Domain. Before Connect all hosts to pool. Time:11/8/12 9:45 AM >2012-11-08 09:45:54,910 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (pool-4-thread-45) [15b20298] START, ValidateStorageServerConnectionVDSCommand(HostName = rhs-gp-srv15.lab.eng.blr.redhat.com, HostId = 12e4f376-29af-11e2-b479-52540035ed48, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = POSIXFS, connectionList = [{ id: 67153a10-6d5c-4c55-a678-075721832deb, connection: rhs-client16.lab.eng.blr.redhat.com:/replicate, iqn: null, vfsType: glusterfs, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 38f632d6 >2012-11-08 09:45:54,910 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (pool-4-thread-47) [68aef695] START, ValidateStorageServerConnectionVDSCommand(HostName = rhs-gp-srv12.lab.eng.blr.redhat.com, HostId = 056a9890-29af-11e2-b5a8-52540035ed48, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = POSIXFS, connectionList = [{ id: 67153a10-6d5c-4c55-a678-075721832deb, connection: rhs-client16.lab.eng.blr.redhat.com:/replicate, iqn: null, vfsType: glusterfs, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 2b839cd1 >2012-11-08 09:45:54,939 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (pool-4-thread-45) [15b20298] FINISH, ValidateStorageServerConnectionVDSCommand, return: {67153a10-6d5c-4c55-a678-075721832deb=0}, log id: 38f632d6 >2012-11-08 09:45:54,939 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (pool-4-thread-47) [68aef695] FINISH, ValidateStorageServerConnectionVDSCommand, return: {67153a10-6d5c-4c55-a678-075721832deb=0}, log id: 2b839cd1 >2012-11-08 09:45:54,939 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (pool-4-thread-45) [15b20298] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2012-11-08 09:45:54,939 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (pool-4-thread-47) [68aef695] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2012-11-08 09:45:54,941 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-4-thread-45) [15b20298] START, ConnectStorageServerVDSCommand(HostName = rhs-gp-srv15.lab.eng.blr.redhat.com, HostId = 12e4f376-29af-11e2-b479-52540035ed48, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = POSIXFS, connectionList = [{ id: 67153a10-6d5c-4c55-a678-075721832deb, connection: rhs-client16.lab.eng.blr.redhat.com:/replicate, iqn: null, vfsType: glusterfs, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 394a2643 >2012-11-08 09:45:54,941 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-4-thread-47) [68aef695] START, ConnectStorageServerVDSCommand(HostName = rhs-gp-srv12.lab.eng.blr.redhat.com, HostId = 056a9890-29af-11e2-b5a8-52540035ed48, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = POSIXFS, connectionList = [{ id: 67153a10-6d5c-4c55-a678-075721832deb, connection: rhs-client16.lab.eng.blr.redhat.com:/replicate, iqn: null, vfsType: glusterfs, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 394a2643 >2012-11-08 09:45:54,973 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-4-thread-47) [68aef695] FINISH, ConnectStorageServerVDSCommand, return: {67153a10-6d5c-4c55-a678-075721832deb=0}, log id: 394a2643 >2012-11-08 09:45:59,183 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-4-thread-45) [15b20298] FINISH, ConnectStorageServerVDSCommand, return: {67153a10-6d5c-4c55-a678-075721832deb=0}, log id: 394a2643 >2012-11-08 09:45:59,184 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-12) [617a9872] START, ActivateStorageDomainVDSCommand( storagePoolId = 1eaedbbb-1c0e-4028-9b0e-adac7e07d25f, ignoreFailoverLimit = false, compatabilityVersion = null, storageDomainId = 2769f0af-c1ca-4cb1-8470-d8cc2e830f7c), log id: 337c68af >2012-11-08 09:45:59,189 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (ajp-/127.0.0.1:8702-12) [749db17] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 1eaedbbb-1c0e-4028-9b0e-adac7e07d25f Type: StoragePool >2012-11-08 09:45:59,244 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (ajp-/127.0.0.1:8702-12) [749db17] hostFromVds::selectedVds - rhs-gp-srv15.lab.eng.blr.redhat.com, spmStatus Unknown_Pool, storage pool RHS-SELF-HEAL >2012-11-08 09:45:59,252 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (ajp-/127.0.0.1:8702-12) [749db17] START, ConnectStoragePoolVDSCommand(HostName = rhs-gp-srv15.lab.eng.blr.redhat.com, HostId = 12e4f376-29af-11e2-b479-52540035ed48, storagePoolId = 1eaedbbb-1c0e-4028-9b0e-adac7e07d25f, vds_spm_id = 2, masterDomainId = 2769f0af-c1ca-4cb1-8470-d8cc2e830f7c, masterVersion = 1), log id: 17294d16 >2012-11-08 09:46:01,987 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (ajp-/127.0.0.1:8702-12) [749db17] FINISH, ConnectStoragePoolVDSCommand, log id: 17294d16 >2012-11-08 09:46:02,015 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (ajp-/127.0.0.1:8702-12) [749db17] hostFromVds::Connected host to pool - selectedVds - rhs-gp-srv15.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-SELF-HEAL >2012-11-08 09:46:02,016 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (ajp-/127.0.0.1:8702-12) [749db17] starting spm on vds rhs-gp-srv15.lab.eng.blr.redhat.com, storage pool RHS-SELF-HEAL, prevId -1, LVER -1 >2012-11-08 09:46:02,017 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (ajp-/127.0.0.1:8702-12) [749db17] START, SpmStartVDSCommand(HostName = rhs-gp-srv15.lab.eng.blr.redhat.com, HostId = 12e4f376-29af-11e2-b479-52540035ed48, storagePoolId = 1eaedbbb-1c0e-4028-9b0e-adac7e07d25f, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7bffdc8e >2012-11-08 09:46:02,053 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (ajp-/127.0.0.1:8702-12) [749db17] spmStart polling started: taskId = 5bff9afd-5a79-4594-a9b7-406dae500f66 >2012-11-08 09:46:04,718 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:46:04,746 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:46:04,748 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:46:04,767 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-42) START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 367d604c >2012-11-08 09:46:04,797 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-42) spmStart polling started: taskId = 34d81028-a7b6-49ac-873d-688b3ece8823 >2012-11-08 09:46:05,837 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-42) Failed in HSMGetTaskStatusVDS method >2012-11-08 09:46:05,837 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-42) Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:46:05,838 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-42) spmStart polling ended: taskId = 34d81028-a7b6-49ac-873d-688b3ece8823 task status = finished >2012-11-08 09:46:05,838 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-42) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:46:05,866 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-42) spmStart polling ended, spm status: Free >2012-11-08 09:46:05,881 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-42) START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=34d81028-a7b6-49ac-873d-688b3ece8823), log id: 33549797 >2012-11-08 09:46:05,904 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-42) FINISH, HSMClearTaskVDSCommand, log id: 33549797 >2012-11-08 09:46:05,904 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-42) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@76ac54d4, log id: 367d604c >2012-11-08 09:46:05,906 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-42) [2bf5672a] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:46:05,922 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [2bf5672a] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:46:06,586 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (ajp-/127.0.0.1:8702-10) [233573a6] START, ValidateStorageServerConnectionVDSCommand(HostName = rhs-gp-srv12.lab.eng.blr.redhat.com, HostId = 056a9890-29af-11e2-b5a8-52540035ed48, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = POSIXFS, connectionList = [{ id: null, connection: rhs-client1.lab.eng.blr.redhat.com:/dis-rep, iqn: null, vfsType: glusterfs, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 6fa73142 >2012-11-08 09:46:06,605 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (ajp-/127.0.0.1:8702-10) [233573a6] FINISH, ValidateStorageServerConnectionVDSCommand, return: {00000000-0000-0000-0000-000000000000=0}, log id: 6fa73142 >2012-11-08 09:46:06,608 INFO [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-10) [233573a6] Running command: AddStorageServerConnectionCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2012-11-08 09:46:06,610 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-10) [233573a6] START, ConnectStorageServerVDSCommand(HostName = rhs-gp-srv12.lab.eng.blr.redhat.com, HostId = 056a9890-29af-11e2-b5a8-52540035ed48, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = POSIXFS, connectionList = [{ id: null, connection: rhs-client1.lab.eng.blr.redhat.com:/dis-rep, iqn: null, vfsType: glusterfs, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 72eff95a >2012-11-08 09:46:10,709 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-10) [233573a6] FINISH, ConnectStorageServerVDSCommand, return: {00000000-0000-0000-0000-000000000000=0}, log id: 72eff95a >2012-11-08 09:46:10,749 WARN [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (ajp-/127.0.0.1:8702-4) [543fdf1c] The message key AddPosixFsStorageDomain is missing from bundles/ExecutionMessages >2012-11-08 09:46:10,766 INFO [org.ovirt.engine.core.bll.storage.AddPosixFsStorageDomainCommand] (ajp-/127.0.0.1:8702-4) [543fdf1c] Running command: AddPosixFsStorageDomainCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2012-11-08 09:46:10,783 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-4) [543fdf1c] START, CreateStorageDomainVDSCommand(HostName = rhs-gp-srv12.lab.eng.blr.redhat.com, HostId = 056a9890-29af-11e2-b5a8-52540035ed48, storageDomain=org.ovirt.engine.core.common.businessentities.storage_domain_static@4e43d735, args=rhs-client1.lab.eng.blr.redhat.com:/dis-rep), log id: 9290f1 >2012-11-08 09:46:14,967 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-4) [543fdf1c] FINISH, CreateStorageDomainVDSCommand, log id: 9290f1 >2012-11-08 09:46:14,968 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStorageDomainStatsVDSCommand] (ajp-/127.0.0.1:8702-4) [543fdf1c] START, GetStorageDomainStatsVDSCommand(HostName = rhs-gp-srv12.lab.eng.blr.redhat.com, HostId = 056a9890-29af-11e2-b5a8-52540035ed48, storageDomainId=e4819bf3-8838-47dc-a92c-c8e9b63c74ca), log id: 172bc360 >2012-11-08 09:46:14,999 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStorageDomainStatsVDSCommand] (ajp-/127.0.0.1:8702-4) [543fdf1c] FINISH, GetStorageDomainStatsVDSCommand, return: org.ovirt.engine.core.common.businessentities.storage_domains@40e72ac1, log id: 172bc360 >2012-11-08 09:46:15,093 WARN [org.ovirt.engine.core.bll.storage.AttachStorageDomainToPoolCommand] (ajp-/127.0.0.1:8702-11) [704a970] CanDoAction of action AttachStorageDomainToPool failed. Reasons:VAR__TYPE__STORAGE__DOMAIN,VAR__ACTION__ATTACH,ACTION_TYPE_FAILED_MASTER_STORAGE_DOMAIN_NOT_ACTIVE >2012-11-08 09:46:15,980 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:46:16,008 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:46:16,011 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:46:16,029 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-73) START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 34d4d8f0 >2012-11-08 09:46:16,051 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-73) spmStart polling started: taskId = 5f1d2d43-e467-4b23-bdbf-bf6ea77c81b3 >2012-11-08 09:46:17,086 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-73) Failed in HSMGetTaskStatusVDS method >2012-11-08 09:46:17,086 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-73) Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:46:17,086 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-73) spmStart polling ended: taskId = 5f1d2d43-e467-4b23-bdbf-bf6ea77c81b3 task status = finished >2012-11-08 09:46:17,086 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-73) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:46:17,117 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-73) spmStart polling ended, spm status: Free >2012-11-08 09:46:17,136 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-73) START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=5f1d2d43-e467-4b23-bdbf-bf6ea77c81b3), log id: 739d264c >2012-11-08 09:46:17,153 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-73) FINISH, HSMClearTaskVDSCommand, log id: 739d264c >2012-11-08 09:46:17,153 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-73) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@523d82e0, log id: 34d4d8f0 >2012-11-08 09:46:17,155 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-73) [445b6a4e] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:46:17,178 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [445b6a4e] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:46:23,604 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (ajp-/127.0.0.1:8702-12) [749db17] spmStart polling ended: taskId = 5bff9afd-5a79-4594-a9b7-406dae500f66 task status = finished >2012-11-08 09:46:23,625 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (ajp-/127.0.0.1:8702-12) [749db17] spmStart polling ended, spm status: SPM >2012-11-08 09:46:23,627 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (ajp-/127.0.0.1:8702-12) [749db17] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv15.lab.eng.blr.redhat.com, HostId = 12e4f376-29af-11e2-b479-52540035ed48, taskId=5bff9afd-5a79-4594-a9b7-406dae500f66), log id: 6af507bc >2012-11-08 09:46:23,654 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (ajp-/127.0.0.1:8702-12) [749db17] FINISH, HSMClearTaskVDSCommand, log id: 6af507bc >2012-11-08 09:46:23,654 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (ajp-/127.0.0.1:8702-12) [749db17] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@60e550d9, log id: 7bffdc8e >2012-11-08 09:46:23,656 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (ajp-/127.0.0.1:8702-12) [749db17] Initialize Irs proxy from vds: 10.70.36.19 >2012-11-08 09:46:23,694 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-12) [749db17] FINISH, ActivateStorageDomainVDSCommand, log id: 337c68af >2012-11-08 09:46:23,694 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp-/127.0.0.1:8702-12) [749db17] ActivateStorage Domain. After Connect all hosts to pool. Time:11/8/12 9:46 AM >2012-11-08 09:46:23,712 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-45) [749db17] START, ConnectStoragePoolVDSCommand(HostName = rhs-gp-srv12.lab.eng.blr.redhat.com, HostId = 056a9890-29af-11e2-b5a8-52540035ed48, storagePoolId = 1eaedbbb-1c0e-4028-9b0e-adac7e07d25f, vds_spm_id = 1, masterDomainId = 2769f0af-c1ca-4cb1-8470-d8cc2e830f7c, masterVersion = 1), log id: ee2f6a5 >2012-11-08 09:46:23,713 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-50) [749db17] START, ConnectStoragePoolVDSCommand(HostName = rhs-gp-srv15.lab.eng.blr.redhat.com, HostId = 12e4f376-29af-11e2-b479-52540035ed48, storagePoolId = 1eaedbbb-1c0e-4028-9b0e-adac7e07d25f, vds_spm_id = 2, masterDomainId = 2769f0af-c1ca-4cb1-8470-d8cc2e830f7c, masterVersion = 1), log id: 6f6edc78 >2012-11-08 09:46:23,741 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-50) [749db17] FINISH, ConnectStoragePoolVDSCommand, log id: 6f6edc78 >2012-11-08 09:46:25,307 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-45) [749db17] FINISH, ConnectStoragePoolVDSCommand, log id: ee2f6a5 >2012-11-08 09:46:25,319 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp-/127.0.0.1:8702-12) [749db17] ActivateStorage Domain. After change storage pool status in vds. Time:11/8/12 9:46 AM >2012-11-08 09:46:27,242 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:46:27,269 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:46:27,272 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:46:27,283 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 32b7901b >2012-11-08 09:46:27,309 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) spmStart polling started: taskId = 1c7ba084-cacc-42e8-a49e-121eccb45a9e >2012-11-08 09:46:28,352 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-16) Failed in HSMGetTaskStatusVDS method >2012-11-08 09:46:28,352 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-16) Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:46:28,353 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) spmStart polling ended: taskId = 1c7ba084-cacc-42e8-a49e-121eccb45a9e task status = finished >2012-11-08 09:46:28,353 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:46:28,381 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) spmStart polling ended, spm status: Free >2012-11-08 09:46:28,392 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-16) START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=1c7ba084-cacc-42e8-a49e-121eccb45a9e), log id: 61da0ed0 >2012-11-08 09:46:28,413 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-16) FINISH, HSMClearTaskVDSCommand, log id: 61da0ed0 >2012-11-08 09:46:28,413 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3302abe0, log id: 32b7901b >2012-11-08 09:46:28,415 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-16) [74621949] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:46:28,443 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [74621949] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:46:38,510 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-36) [143e331d] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:46:38,533 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-36) [143e331d] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:46:38,535 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-36) [143e331d] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:46:38,554 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-36) [143e331d] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 400d6a0c >2012-11-08 09:46:38,578 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-36) [143e331d] spmStart polling started: taskId = 8cf2d239-ff4e-4dea-8e42-954a9390b57d >2012-11-08 09:46:39,622 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-36) [143e331d] Failed in HSMGetTaskStatusVDS method >2012-11-08 09:46:39,622 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-36) [143e331d] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:46:39,622 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-36) [143e331d] spmStart polling ended: taskId = 8cf2d239-ff4e-4dea-8e42-954a9390b57d task status = finished >2012-11-08 09:46:39,622 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-36) [143e331d] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:46:39,649 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-36) [143e331d] spmStart polling ended, spm status: Free >2012-11-08 09:46:39,666 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-36) [143e331d] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=8cf2d239-ff4e-4dea-8e42-954a9390b57d), log id: 652c5a32 >2012-11-08 09:46:39,686 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-36) [143e331d] FINISH, HSMClearTaskVDSCommand, log id: 652c5a32 >2012-11-08 09:46:39,686 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-36) [143e331d] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@42d5bc82, log id: 400d6a0c >2012-11-08 09:46:39,689 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-36) [5e4b1d15] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:46:39,713 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-36) [5e4b1d15] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:46:49,779 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [401625b4] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:46:49,807 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [401625b4] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:46:49,810 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [401625b4] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:46:49,827 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-74) [401625b4] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 414cde9d >2012-11-08 09:46:49,851 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-74) [401625b4] spmStart polling started: taskId = 46c5b238-dc19-40ec-ac2f-ed6c1c12810f >2012-11-08 09:46:50,884 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-74) [401625b4] Failed in HSMGetTaskStatusVDS method >2012-11-08 09:46:50,884 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-74) [401625b4] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:46:50,884 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-74) [401625b4] spmStart polling ended: taskId = 46c5b238-dc19-40ec-ac2f-ed6c1c12810f task status = finished >2012-11-08 09:46:50,884 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-74) [401625b4] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:46:50,913 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-74) [401625b4] spmStart polling ended, spm status: Free >2012-11-08 09:46:50,924 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-74) [401625b4] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=46c5b238-dc19-40ec-ac2f-ed6c1c12810f), log id: 3fae06e3 >2012-11-08 09:46:50,945 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-74) [401625b4] FINISH, HSMClearTaskVDSCommand, log id: 3fae06e3 >2012-11-08 09:46:50,945 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-74) [401625b4] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@323f970f, log id: 414cde9d >2012-11-08 09:46:50,947 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-74) [6cd6738e] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:46:50,974 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [6cd6738e] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:47:01,037 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [676866c2] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:47:01,066 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [676866c2] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:47:01,069 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [676866c2] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:47:01,086 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [676866c2] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3f8e8b59 >2012-11-08 09:47:01,116 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [676866c2] spmStart polling started: taskId = bc9e58ff-9a94-4210-afef-0004cecf5359 >2012-11-08 09:47:02,160 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-14) [676866c2] Failed in HSMGetTaskStatusVDS method >2012-11-08 09:47:02,161 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-14) [676866c2] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:47:02,161 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [676866c2] spmStart polling ended: taskId = bc9e58ff-9a94-4210-afef-0004cecf5359 task status = finished >2012-11-08 09:47:02,161 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [676866c2] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:47:02,188 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [676866c2] spmStart polling ended, spm status: Free >2012-11-08 09:47:02,201 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-14) [676866c2] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=bc9e58ff-9a94-4210-afef-0004cecf5359), log id: 3468220e >2012-11-08 09:47:02,220 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-14) [676866c2] FINISH, HSMClearTaskVDSCommand, log id: 3468220e >2012-11-08 09:47:02,220 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [676866c2] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2159a5b9, log id: 3f8e8b59 >2012-11-08 09:47:02,222 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-14) [48f25ca] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:47:02,242 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [48f25ca] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:47:08,329 INFO [org.ovirt.engine.core.bll.SetVmTicketCommand] (ajp-/127.0.0.1:8702-9) [2d05bca0] Running command: SetVmTicketCommand internal: false. Entities affected : ID: 7fa6f92f-755f-44c3-8f35-b4c56410c87b Type: VM >2012-11-08 09:47:08,352 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp-/127.0.0.1:8702-9) [2d05bca0] START, SetVmTicketVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=7fa6f92f-755f-44c3-8f35-b4c56410c87b, ticket=/WjChJKdz75X, validTime=120,m userName=admin@internal, userId=fdfc627c-d875-11e0-90f0-83df133b58cc), log id: 63fbf660 >2012-11-08 09:47:08,383 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp-/127.0.0.1:8702-9) [2d05bca0] FINISH, SetVmTicketVDSCommand, log id: 63fbf660 >2012-11-08 09:47:12,298 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-45) [78ee57dc] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:47:12,326 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-45) [78ee57dc] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:47:12,330 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-45) [78ee57dc] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:47:12,346 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-45) [78ee57dc] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4b9a1958 >2012-11-08 09:47:12,369 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-45) [78ee57dc] spmStart polling started: taskId = e133b2fe-cfda-4f77-8765-dc5e1cc8d69b >2012-11-08 09:47:13,410 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-45) [78ee57dc] Failed in HSMGetTaskStatusVDS method >2012-11-08 09:47:13,410 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-45) [78ee57dc] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:47:13,410 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-45) [78ee57dc] spmStart polling ended: taskId = e133b2fe-cfda-4f77-8765-dc5e1cc8d69b task status = finished >2012-11-08 09:47:13,410 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-45) [78ee57dc] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:47:13,438 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-45) [78ee57dc] spmStart polling ended, spm status: Free >2012-11-08 09:47:13,451 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-45) [78ee57dc] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=e133b2fe-cfda-4f77-8765-dc5e1cc8d69b), log id: 3bfd66fc >2012-11-08 09:47:13,473 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-45) [78ee57dc] FINISH, HSMClearTaskVDSCommand, log id: 3bfd66fc >2012-11-08 09:47:13,473 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-45) [78ee57dc] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@79744928, log id: 4b9a1958 >2012-11-08 09:47:13,475 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-45) [416f1f63] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:47:13,499 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-45) [416f1f63] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:47:20,706 INFO [org.ovirt.engine.core.bll.storage.RemoveStorageDomainCommand] (ajp-/127.0.0.1:8702-1) [29adf3a0] Lock Acquired to object EngineLock [exclusiveLocks= key: e4819bf3-8838-47dc-a92c-c8e9b63c74ca value: STORAGE >, sharedLocks= ] >2012-11-08 09:47:20,709 INFO [org.ovirt.engine.core.bll.storage.RemoveStorageDomainCommand] (ajp-/127.0.0.1:8702-1) [29adf3a0] Running command: RemoveStorageDomainCommand internal: false. Entities affected : ID: e4819bf3-8838-47dc-a92c-c8e9b63c74ca Type: Storage >2012-11-08 09:47:20,713 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (ajp-/127.0.0.1:8702-1) [3a732395] START, ValidateStorageServerConnectionVDSCommand(HostName = rhs-arch-srv2.lab.eng.blr.redhat.com, HostId = cf017816-29a2-11e2-949e-52540035ed48, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = POSIXFS, connectionList = [{ id: b638d0f2-aa22-4e6d-b444-3023a393ef61, connection: rhs-client1.lab.eng.blr.redhat.com:/dis-rep, iqn: null, vfsType: glusterfs, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 549905a0 >2012-11-08 09:47:20,733 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (ajp-/127.0.0.1:8702-1) [3a732395] FINISH, ValidateStorageServerConnectionVDSCommand, return: {b638d0f2-aa22-4e6d-b444-3023a393ef61=0}, log id: 549905a0 >2012-11-08 09:47:20,733 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (ajp-/127.0.0.1:8702-1) [3a732395] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2012-11-08 09:47:20,735 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-1) [3a732395] START, ConnectStorageServerVDSCommand(HostName = rhs-arch-srv2.lab.eng.blr.redhat.com, HostId = cf017816-29a2-11e2-949e-52540035ed48, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = POSIXFS, connectionList = [{ id: b638d0f2-aa22-4e6d-b444-3023a393ef61, connection: rhs-client1.lab.eng.blr.redhat.com:/dis-rep, iqn: null, vfsType: glusterfs, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 225657a6 >2012-11-08 09:47:23,562 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) [5a399fab] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:47:23,587 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) [5a399fab] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:47:23,590 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) [5a399fab] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:47:23,606 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-65) [5a399fab] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 52a72f04 >2012-11-08 09:47:23,628 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-65) [5a399fab] spmStart polling started: taskId = be56a13a-18d3-43bb-93ca-37b6ed6d5c9d >2012-11-08 09:47:24,659 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-65) [5a399fab] Failed in HSMGetTaskStatusVDS method >2012-11-08 09:47:24,659 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-65) [5a399fab] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:47:24,659 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-65) [5a399fab] spmStart polling ended: taskId = be56a13a-18d3-43bb-93ca-37b6ed6d5c9d task status = finished >2012-11-08 09:47:24,659 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-65) [5a399fab] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:47:24,687 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-65) [5a399fab] spmStart polling ended, spm status: Free >2012-11-08 09:47:24,699 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-65) [5a399fab] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=be56a13a-18d3-43bb-93ca-37b6ed6d5c9d), log id: 3628c5b2 >2012-11-08 09:47:24,718 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-65) [5a399fab] FINISH, HSMClearTaskVDSCommand, log id: 3628c5b2 >2012-11-08 09:47:24,718 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-65) [5a399fab] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@5957ba54, log id: 52a72f04 >2012-11-08 09:47:24,720 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-65) [e8a0e9f] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:47:24,736 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) [e8a0e9f] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:47:24,869 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-1) [3a732395] FINISH, ConnectStorageServerVDSCommand, return: {b638d0f2-aa22-4e6d-b444-3023a393ef61=0}, log id: 225657a6 >2012-11-08 09:47:24,872 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FormatStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-1) [3a732395] START, FormatStorageDomainVDSCommand(HostName = rhs-arch-srv2.lab.eng.blr.redhat.com, HostId = cf017816-29a2-11e2-949e-52540035ed48, storageDomainId=e4819bf3-8838-47dc-a92c-c8e9b63c74ca), log id: 49dbcc72 >2012-11-08 09:47:29,627 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FormatStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-1) [3a732395] FINISH, FormatStorageDomainVDSCommand, log id: 49dbcc72 >2012-11-08 09:47:29,629 INFO [org.ovirt.engine.core.bll.storage.RemoveStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-1) [246060dd] Running command: RemoveStorageServerConnectionCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2012-11-08 09:47:29,630 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-1) [246060dd] START, DisconnectStorageServerVDSCommand(HostName = rhs-arch-srv2.lab.eng.blr.redhat.com, HostId = cf017816-29a2-11e2-949e-52540035ed48, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = POSIXFS, connectionList = [{ id: b638d0f2-aa22-4e6d-b444-3023a393ef61, connection: rhs-client1.lab.eng.blr.redhat.com:/dis-rep, iqn: null, vfsType: glusterfs, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 7c07c65e >2012-11-08 09:47:31,768 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-1) [246060dd] FINISH, DisconnectStorageServerVDSCommand, return: {b638d0f2-aa22-4e6d-b444-3023a393ef61=0}, log id: 7c07c65e >2012-11-08 09:47:31,776 INFO [org.ovirt.engine.core.bll.storage.RemoveStorageDomainCommand] (ajp-/127.0.0.1:8702-1) [246060dd] Lock freed to object EngineLock [exclusiveLocks= key: e4819bf3-8838-47dc-a92c-c8e9b63c74ca value: STORAGE >, sharedLocks= ] >2012-11-08 09:47:34,783 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [74621949] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:47:34,813 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [74621949] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:47:34,816 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [74621949] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:47:34,834 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [74621949] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 422e12c9 >2012-11-08 09:47:34,856 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [74621949] spmStart polling started: taskId = 0f79dd14-4b6f-4196-a9d6-5e45593e97af >2012-11-08 09:47:35,887 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-16) [74621949] Failed in HSMGetTaskStatusVDS method >2012-11-08 09:47:35,888 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-16) [74621949] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:47:35,888 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [74621949] spmStart polling ended: taskId = 0f79dd14-4b6f-4196-a9d6-5e45593e97af task status = finished >2012-11-08 09:47:35,888 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [74621949] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:47:35,919 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [74621949] spmStart polling ended, spm status: Free >2012-11-08 09:47:35,935 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-16) [74621949] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=0f79dd14-4b6f-4196-a9d6-5e45593e97af), log id: 697241ca >2012-11-08 09:47:35,953 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-16) [74621949] FINISH, HSMClearTaskVDSCommand, log id: 697241ca >2012-11-08 09:47:35,953 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [74621949] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4c3d6adb, log id: 422e12c9 >2012-11-08 09:47:35,956 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-16) [543276f7] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:47:35,979 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [543276f7] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:47:46,029 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-40) [46f2edae] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:47:46,056 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-40) [46f2edae] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:47:46,059 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-40) [46f2edae] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:47:46,075 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-40) [46f2edae] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4494a87f >2012-11-08 09:47:46,103 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-40) [46f2edae] spmStart polling started: taskId = a5ea41af-0ab2-47a7-80e2-86e6bde33956 >2012-11-08 09:47:47,143 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-40) [46f2edae] Failed in HSMGetTaskStatusVDS method >2012-11-08 09:47:47,143 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-40) [46f2edae] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:47:47,143 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-40) [46f2edae] spmStart polling ended: taskId = a5ea41af-0ab2-47a7-80e2-86e6bde33956 task status = finished >2012-11-08 09:47:47,143 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-40) [46f2edae] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:47:47,174 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-40) [46f2edae] spmStart polling ended, spm status: Free >2012-11-08 09:47:47,187 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-40) [46f2edae] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=a5ea41af-0ab2-47a7-80e2-86e6bde33956), log id: 22b7a2e2 >2012-11-08 09:47:47,206 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-40) [46f2edae] FINISH, HSMClearTaskVDSCommand, log id: 22b7a2e2 >2012-11-08 09:47:47,206 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-40) [46f2edae] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4626b53c, log id: 4494a87f >2012-11-08 09:47:47,208 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-40) [17449a1f] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:47:47,231 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-40) [17449a1f] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:47:57,101 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (ajp-/127.0.0.1:8702-11) [6d324132] START, ValidateStorageServerConnectionVDSCommand(HostName = rhs-gp-srv15.lab.eng.blr.redhat.com, HostId = 12e4f376-29af-11e2-b479-52540035ed48, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = POSIXFS, connectionList = [{ id: null, connection: rhs-client1.lab.eng.blr.redhat.com:/dis-rep, iqn: null, vfsType: glusterfs, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 626cb50a >2012-11-08 09:47:57,124 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (ajp-/127.0.0.1:8702-11) [6d324132] FINISH, ValidateStorageServerConnectionVDSCommand, return: {00000000-0000-0000-0000-000000000000=0}, log id: 626cb50a >2012-11-08 09:47:57,128 INFO [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-11) [6d324132] Running command: AddStorageServerConnectionCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2012-11-08 09:47:57,131 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-11) [6d324132] START, ConnectStorageServerVDSCommand(HostName = rhs-gp-srv15.lab.eng.blr.redhat.com, HostId = 12e4f376-29af-11e2-b479-52540035ed48, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = POSIXFS, connectionList = [{ id: null, connection: rhs-client1.lab.eng.blr.redhat.com:/dis-rep, iqn: null, vfsType: glusterfs, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 727c1d51 >2012-11-08 09:47:57,295 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [6cd6738e] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:47:57,324 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [6cd6738e] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:47:57,326 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [6cd6738e] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:47:57,338 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-74) [6cd6738e] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3eedf2eb >2012-11-08 09:47:57,361 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-74) [6cd6738e] spmStart polling started: taskId = 22ad8a62-b986-4feb-bf13-4352b545f4b3 >2012-11-08 09:47:58,404 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-74) [6cd6738e] Failed in HSMGetTaskStatusVDS method >2012-11-08 09:47:58,404 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-74) [6cd6738e] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:47:58,404 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-74) [6cd6738e] spmStart polling ended: taskId = 22ad8a62-b986-4feb-bf13-4352b545f4b3 task status = finished >2012-11-08 09:47:58,404 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-74) [6cd6738e] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:47:58,434 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-74) [6cd6738e] spmStart polling ended, spm status: Free >2012-11-08 09:47:58,448 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-74) [6cd6738e] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=22ad8a62-b986-4feb-bf13-4352b545f4b3), log id: 60e7926e >2012-11-08 09:47:58,465 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-74) [6cd6738e] FINISH, HSMClearTaskVDSCommand, log id: 60e7926e >2012-11-08 09:47:58,465 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-74) [6cd6738e] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@33ea576, log id: 3eedf2eb >2012-11-08 09:47:58,467 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-74) [5e0e7a4] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:47:58,482 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [5e0e7a4] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:48:01,235 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-11) [6d324132] FINISH, ConnectStorageServerVDSCommand, return: {00000000-0000-0000-0000-000000000000=0}, log id: 727c1d51 >2012-11-08 09:48:01,271 WARN [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (ajp-/127.0.0.1:8702-3) [2f8e7ebb] The message key AddPosixFsStorageDomain is missing from bundles/ExecutionMessages >2012-11-08 09:48:01,285 INFO [org.ovirt.engine.core.bll.storage.AddPosixFsStorageDomainCommand] (ajp-/127.0.0.1:8702-3) [2f8e7ebb] Running command: AddPosixFsStorageDomainCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2012-11-08 09:48:01,290 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-3) [2f8e7ebb] START, CreateStorageDomainVDSCommand(HostName = rhs-gp-srv15.lab.eng.blr.redhat.com, HostId = 12e4f376-29af-11e2-b479-52540035ed48, storageDomain=org.ovirt.engine.core.common.businessentities.storage_domain_static@4218ed0d, args=rhs-client1.lab.eng.blr.redhat.com:/dis-rep), log id: 4cb694f3 >2012-11-08 09:48:04,593 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-3) [2f8e7ebb] FINISH, CreateStorageDomainVDSCommand, log id: 4cb694f3 >2012-11-08 09:48:04,595 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStorageDomainStatsVDSCommand] (ajp-/127.0.0.1:8702-3) [2f8e7ebb] START, GetStorageDomainStatsVDSCommand(HostName = rhs-gp-srv15.lab.eng.blr.redhat.com, HostId = 12e4f376-29af-11e2-b479-52540035ed48, storageDomainId=61dd7e2a-8df1-4e72-8421-f7441c9d7665), log id: 442fb239 >2012-11-08 09:48:04,624 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStorageDomainStatsVDSCommand] (ajp-/127.0.0.1:8702-3) [2f8e7ebb] FINISH, GetStorageDomainStatsVDSCommand, return: org.ovirt.engine.core.common.businessentities.storage_domains@b133de01, log id: 442fb239 >2012-11-08 09:48:04,718 INFO [org.ovirt.engine.core.bll.storage.AttachStorageDomainToPoolCommand] (ajp-/127.0.0.1:8702-14) [6dc3c2b3] Running command: AttachStorageDomainToPoolCommand internal: false. Entities affected : ID: 61dd7e2a-8df1-4e72-8421-f7441c9d7665 Type: Storage >2012-11-08 09:48:04,747 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (pool-4-thread-45) [4ad26f1e] START, ValidateStorageServerConnectionVDSCommand(HostName = rhs-gp-srv15.lab.eng.blr.redhat.com, HostId = 12e4f376-29af-11e2-b479-52540035ed48, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = POSIXFS, connectionList = [{ id: a3e77e09-8933-421f-900e-47f45e401f5a, connection: rhs-client1.lab.eng.blr.redhat.com:/dis-rep, iqn: null, vfsType: glusterfs, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 3ef78c29 >2012-11-08 09:48:04,751 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (pool-4-thread-47) [223dc552] START, ValidateStorageServerConnectionVDSCommand(HostName = rhs-gp-srv12.lab.eng.blr.redhat.com, HostId = 056a9890-29af-11e2-b5a8-52540035ed48, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = POSIXFS, connectionList = [{ id: a3e77e09-8933-421f-900e-47f45e401f5a, connection: rhs-client1.lab.eng.blr.redhat.com:/dis-rep, iqn: null, vfsType: glusterfs, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 7ef67fa4 >2012-11-08 09:48:04,773 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (pool-4-thread-45) [4ad26f1e] FINISH, ValidateStorageServerConnectionVDSCommand, return: {a3e77e09-8933-421f-900e-47f45e401f5a=0}, log id: 3ef78c29 >2012-11-08 09:48:04,773 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (pool-4-thread-47) [223dc552] FINISH, ValidateStorageServerConnectionVDSCommand, return: {a3e77e09-8933-421f-900e-47f45e401f5a=0}, log id: 7ef67fa4 >2012-11-08 09:48:04,773 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (pool-4-thread-47) [223dc552] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2012-11-08 09:48:04,773 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (pool-4-thread-45) [4ad26f1e] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2012-11-08 09:48:04,774 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-4-thread-45) [4ad26f1e] START, ConnectStorageServerVDSCommand(HostName = rhs-gp-srv15.lab.eng.blr.redhat.com, HostId = 12e4f376-29af-11e2-b479-52540035ed48, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = POSIXFS, connectionList = [{ id: a3e77e09-8933-421f-900e-47f45e401f5a, connection: rhs-client1.lab.eng.blr.redhat.com:/dis-rep, iqn: null, vfsType: glusterfs, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 1decf17 >2012-11-08 09:48:04,774 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-4-thread-47) [223dc552] START, ConnectStorageServerVDSCommand(HostName = rhs-gp-srv12.lab.eng.blr.redhat.com, HostId = 056a9890-29af-11e2-b5a8-52540035ed48, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = POSIXFS, connectionList = [{ id: a3e77e09-8933-421f-900e-47f45e401f5a, connection: rhs-client1.lab.eng.blr.redhat.com:/dis-rep, iqn: null, vfsType: glusterfs, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 4aedeff6 >2012-11-08 09:48:04,794 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-4-thread-47) [223dc552] FINISH, ConnectStorageServerVDSCommand, return: {a3e77e09-8933-421f-900e-47f45e401f5a=0}, log id: 4aedeff6 >2012-11-08 09:48:04,798 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-4-thread-45) [4ad26f1e] FINISH, ConnectStorageServerVDSCommand, return: {a3e77e09-8933-421f-900e-47f45e401f5a=0}, log id: 1decf17 >2012-11-08 09:48:04,799 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.AttachStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-14) [6dc3c2b3] START, AttachStorageDomainVDSCommand( storagePoolId = 1eaedbbb-1c0e-4028-9b0e-adac7e07d25f, ignoreFailoverLimit = false, compatabilityVersion = null, storageDomainId = 61dd7e2a-8df1-4e72-8421-f7441c9d7665), log id: 7ba00cfb >2012-11-08 09:48:08,536 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) [17db7eb0] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:48:08,563 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) [17db7eb0] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:48:08,566 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) [17db7eb0] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:48:08,583 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-10) [17db7eb0] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 357b2842 >2012-11-08 09:48:08,612 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-10) [17db7eb0] spmStart polling started: taskId = 18960449-25d8-4867-b760-6a7861262f90 >2012-11-08 09:48:09,650 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-10) [17db7eb0] Failed in HSMGetTaskStatusVDS method >2012-11-08 09:48:09,650 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-10) [17db7eb0] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:48:09,650 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-10) [17db7eb0] spmStart polling ended: taskId = 18960449-25d8-4867-b760-6a7861262f90 task status = finished >2012-11-08 09:48:09,650 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-10) [17db7eb0] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:48:09,677 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-10) [17db7eb0] spmStart polling ended, spm status: Free >2012-11-08 09:48:09,694 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-10) [17db7eb0] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=18960449-25d8-4867-b760-6a7861262f90), log id: 5caa7982 >2012-11-08 09:48:09,711 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-10) [17db7eb0] FINISH, HSMClearTaskVDSCommand, log id: 5caa7982 >2012-11-08 09:48:09,711 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-10) [17db7eb0] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3c0f7555, log id: 357b2842 >2012-11-08 09:48:09,714 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-10) [1ae03e41] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:48:09,736 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) [1ae03e41] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:48:19,789 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-45) [416f1f63] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:48:19,820 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-45) [416f1f63] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:48:19,823 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-45) [416f1f63] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:48:19,841 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-45) [416f1f63] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 53b62037 >2012-11-08 09:48:19,863 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-45) [416f1f63] spmStart polling started: taskId = 3871e813-2a91-4928-a5c3-e7cd55983b96 >2012-11-08 09:48:20,898 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-45) [416f1f63] Failed in HSMGetTaskStatusVDS method >2012-11-08 09:48:20,898 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-45) [416f1f63] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:48:20,898 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-45) [416f1f63] spmStart polling ended: taskId = 3871e813-2a91-4928-a5c3-e7cd55983b96 task status = finished >2012-11-08 09:48:20,898 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-45) [416f1f63] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:48:20,925 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-45) [416f1f63] spmStart polling ended, spm status: Free >2012-11-08 09:48:20,943 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-45) [416f1f63] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=3871e813-2a91-4928-a5c3-e7cd55983b96), log id: 620c0b57 >2012-11-08 09:48:20,967 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-45) [416f1f63] FINISH, HSMClearTaskVDSCommand, log id: 620c0b57 >2012-11-08 09:48:20,968 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-45) [416f1f63] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4bcaf0b, log id: 53b62037 >2012-11-08 09:48:20,970 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-45) [7f87fd9a] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:48:20,995 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-45) [7f87fd9a] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:48:28,965 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.AttachStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-14) [6dc3c2b3] FINISH, AttachStorageDomainVDSCommand, log id: 7ba00cfb >2012-11-08 09:48:31,051 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) [e8a0e9f] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:48:31,080 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) [e8a0e9f] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:48:31,083 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) [e8a0e9f] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:48:31,101 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-65) [e8a0e9f] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 78a5f5a0 >2012-11-08 09:48:31,126 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-65) [e8a0e9f] spmStart polling started: taskId = 85833320-2902-46fd-b438-a554716a79a6 >2012-11-08 09:48:32,160 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-65) [e8a0e9f] Failed in HSMGetTaskStatusVDS method >2012-11-08 09:48:32,160 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-65) [e8a0e9f] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:48:32,160 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-65) [e8a0e9f] spmStart polling ended: taskId = 85833320-2902-46fd-b438-a554716a79a6 task status = finished >2012-11-08 09:48:32,160 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-65) [e8a0e9f] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:48:32,191 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-65) [e8a0e9f] spmStart polling ended, spm status: Free >2012-11-08 09:48:32,210 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-65) [e8a0e9f] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=85833320-2902-46fd-b438-a554716a79a6), log id: 6d44d167 >2012-11-08 09:48:32,227 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-65) [e8a0e9f] FINISH, HSMClearTaskVDSCommand, log id: 6d44d167 >2012-11-08 09:48:32,227 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-65) [e8a0e9f] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@410ff93c, log id: 78a5f5a0 >2012-11-08 09:48:32,229 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-65) [7fb3f382] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:48:32,252 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) [7fb3f382] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:48:42,313 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [48f25ca] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:48:42,336 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [48f25ca] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:48:42,339 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [48f25ca] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:48:42,359 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [48f25ca] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 62a61e05 >2012-11-08 09:48:42,390 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [48f25ca] spmStart polling started: taskId = be78723f-627c-464f-a0e3-58a33723e077 >2012-11-08 09:48:43,427 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-14) [48f25ca] Failed in HSMGetTaskStatusVDS method >2012-11-08 09:48:43,427 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-14) [48f25ca] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:48:43,427 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [48f25ca] spmStart polling ended: taskId = be78723f-627c-464f-a0e3-58a33723e077 task status = finished >2012-11-08 09:48:43,427 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [48f25ca] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:48:43,462 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [48f25ca] spmStart polling ended, spm status: Free >2012-11-08 09:48:43,475 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-14) [48f25ca] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=be78723f-627c-464f-a0e3-58a33723e077), log id: 2e9c9f87 >2012-11-08 09:48:43,498 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-14) [48f25ca] FINISH, HSMClearTaskVDSCommand, log id: 2e9c9f87 >2012-11-08 09:48:43,498 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [48f25ca] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2cad6ff9, log id: 62a61e05 >2012-11-08 09:48:43,500 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-14) [2e925b59] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:48:43,538 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [2e925b59] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:48:53,599 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-47) hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:48:53,623 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-47) SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:48:53,626 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-47) starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:48:53,642 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-47) START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 458a952e >2012-11-08 09:48:53,671 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-47) spmStart polling started: taskId = 0fa21749-b20a-4574-ac9f-7165bdbfa2ae >2012-11-08 09:48:54,702 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-47) Failed in HSMGetTaskStatusVDS method >2012-11-08 09:48:54,702 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-47) Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:48:54,702 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-47) spmStart polling ended: taskId = 0fa21749-b20a-4574-ac9f-7165bdbfa2ae task status = finished >2012-11-08 09:48:54,702 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-47) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:48:54,731 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-47) spmStart polling ended, spm status: Free >2012-11-08 09:48:54,750 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-47) START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=0fa21749-b20a-4574-ac9f-7165bdbfa2ae), log id: 4ffe2023 >2012-11-08 09:48:54,775 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-47) FINISH, HSMClearTaskVDSCommand, log id: 4ffe2023 >2012-11-08 09:48:54,775 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-47) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@76a7441f, log id: 458a952e >2012-11-08 09:48:54,776 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-47) [5f698c14] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:48:54,790 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-47) [5f698c14] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:49:04,853 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [6193d74c] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:49:04,880 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [6193d74c] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:49:04,882 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [6193d74c] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:49:04,892 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-71) [6193d74c] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7ea5fcd2 >2012-11-08 09:49:04,922 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-71) [6193d74c] spmStart polling started: taskId = 66c4b9a3-870b-4ae9-9714-d9a51e1d6f6c >2012-11-08 09:49:05,953 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-71) [6193d74c] Failed in HSMGetTaskStatusVDS method >2012-11-08 09:49:05,953 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-71) [6193d74c] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:49:05,953 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-71) [6193d74c] spmStart polling ended: taskId = 66c4b9a3-870b-4ae9-9714-d9a51e1d6f6c task status = finished >2012-11-08 09:49:05,953 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-71) [6193d74c] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:49:05,981 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-71) [6193d74c] spmStart polling ended, spm status: Free >2012-11-08 09:49:05,999 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-71) [6193d74c] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=66c4b9a3-870b-4ae9-9714-d9a51e1d6f6c), log id: 6902c001 >2012-11-08 09:49:06,016 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-71) [6193d74c] FINISH, HSMClearTaskVDSCommand, log id: 6902c001 >2012-11-08 09:49:06,016 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-71) [6193d74c] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@338bb783, log id: 7ea5fcd2 >2012-11-08 09:49:06,017 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-71) [19bd13e5] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:49:06,040 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [19bd13e5] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:49:16,095 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-11) [25cff2cc] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:49:16,128 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-11) [25cff2cc] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:49:16,130 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-11) [25cff2cc] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:49:16,143 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-11) [25cff2cc] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 17bd468c >2012-11-08 09:49:16,166 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-11) [25cff2cc] spmStart polling started: taskId = 7ad5cedf-c2ec-4f1b-87f5-30420846849b >2012-11-08 09:49:17,196 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-11) [25cff2cc] Failed in HSMGetTaskStatusVDS method >2012-11-08 09:49:17,196 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-11) [25cff2cc] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:49:17,196 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-11) [25cff2cc] spmStart polling ended: taskId = 7ad5cedf-c2ec-4f1b-87f5-30420846849b task status = finished >2012-11-08 09:49:17,196 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-11) [25cff2cc] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:49:17,223 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-11) [25cff2cc] spmStart polling ended, spm status: Free >2012-11-08 09:49:17,235 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-11) [25cff2cc] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=7ad5cedf-c2ec-4f1b-87f5-30420846849b), log id: 30f2b936 >2012-11-08 09:49:17,254 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-11) [25cff2cc] FINISH, HSMClearTaskVDSCommand, log id: 30f2b936 >2012-11-08 09:49:17,254 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-11) [25cff2cc] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@f5da155, log id: 17bd468c >2012-11-08 09:49:17,255 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-11) [4c0adb54] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:49:17,276 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-11) [4c0adb54] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:49:27,335 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-29) [23c6b14f] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:49:27,362 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-29) [23c6b14f] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:49:27,365 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-29) [23c6b14f] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:49:27,376 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-29) [23c6b14f] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 16e67081 >2012-11-08 09:49:27,399 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-29) [23c6b14f] spmStart polling started: taskId = c46df7e3-9956-4cb1-894f-700c31b92784 >2012-11-08 09:49:28,436 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-29) [23c6b14f] Failed in HSMGetTaskStatusVDS method >2012-11-08 09:49:28,436 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-29) [23c6b14f] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:49:28,436 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-29) [23c6b14f] spmStart polling ended: taskId = c46df7e3-9956-4cb1-894f-700c31b92784 task status = finished >2012-11-08 09:49:28,436 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-29) [23c6b14f] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:49:28,464 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-29) [23c6b14f] spmStart polling ended, spm status: Free >2012-11-08 09:49:28,480 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-29) [23c6b14f] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=c46df7e3-9956-4cb1-894f-700c31b92784), log id: 1c4af74 >2012-11-08 09:49:28,497 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-29) [23c6b14f] FINISH, HSMClearTaskVDSCommand, log id: 1c4af74 >2012-11-08 09:49:28,497 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-29) [23c6b14f] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@17dae994, log id: 16e67081 >2012-11-08 09:49:28,499 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-29) [251cffc8] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:49:28,523 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-29) [251cffc8] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:49:38,590 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [712c9206] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:49:38,618 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [712c9206] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:49:38,622 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [712c9206] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:49:38,639 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-82) [712c9206] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 179ab32d >2012-11-08 09:49:38,672 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-82) [712c9206] spmStart polling started: taskId = e89b757f-4886-4dba-9ba4-be013942fa02 >2012-11-08 09:49:39,712 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-82) [712c9206] Failed in HSMGetTaskStatusVDS method >2012-11-08 09:49:39,712 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-82) [712c9206] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:49:39,712 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-82) [712c9206] spmStart polling ended: taskId = e89b757f-4886-4dba-9ba4-be013942fa02 task status = finished >2012-11-08 09:49:39,712 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-82) [712c9206] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:49:39,741 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-82) [712c9206] spmStart polling ended, spm status: Free >2012-11-08 09:49:39,758 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-82) [712c9206] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=e89b757f-4886-4dba-9ba4-be013942fa02), log id: 1352f04a >2012-11-08 09:49:39,783 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-82) [712c9206] FINISH, HSMClearTaskVDSCommand, log id: 1352f04a >2012-11-08 09:49:39,784 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-82) [712c9206] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2a1d942f, log id: 179ab32d >2012-11-08 09:49:39,786 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-82) [7bebaf42] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:49:39,811 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [7bebaf42] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:49:49,873 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [2e925b59] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:49:49,899 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [2e925b59] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:49:49,902 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [2e925b59] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:49:49,919 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [2e925b59] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6a98a6d2 >2012-11-08 09:49:49,944 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [2e925b59] spmStart polling started: taskId = 9a8d4104-a696-47a7-86fc-a573002e89e3 >2012-11-08 09:49:50,978 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-14) [2e925b59] Failed in HSMGetTaskStatusVDS method >2012-11-08 09:49:50,978 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-14) [2e925b59] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:49:50,978 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [2e925b59] spmStart polling ended: taskId = 9a8d4104-a696-47a7-86fc-a573002e89e3 task status = finished >2012-11-08 09:49:50,978 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [2e925b59] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:49:51,007 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [2e925b59] spmStart polling ended, spm status: Free >2012-11-08 09:49:51,021 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-14) [2e925b59] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=9a8d4104-a696-47a7-86fc-a573002e89e3), log id: 899e7c4 >2012-11-08 09:49:51,039 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-14) [2e925b59] FINISH, HSMClearTaskVDSCommand, log id: 899e7c4 >2012-11-08 09:49:51,041 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [2e925b59] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@303ef945, log id: 6a98a6d2 >2012-11-08 09:49:51,043 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-14) [765f39c1] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:49:51,064 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [765f39c1] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:49:58,053 INFO [org.ovirt.engine.core.bll.LogoutUserCommand] (ajp-/127.0.0.1:8702-14) [41df93fd] Running command: LogoutUserCommand internal: false. >2012-11-08 09:49:58,498 INFO [org.ovirt.engine.core.bll.LogoutUserCommand] (ajp-/127.0.0.1:8702-15) [bf401af] Running command: LogoutUserCommand internal: false. >2012-11-08 09:49:58,812 INFO [org.ovirt.engine.core.bll.LogoutUserCommand] (ajp-/127.0.0.1:8702-12) [7fc17fa3] Running command: LogoutUserCommand internal: false. >2012-11-08 09:50:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-33) [5e994d3f] Autorecovering hosts is disabled, skipping >2012-11-08 09:50:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-33) [5e994d3f] Autorecovering storage domains is disabled, skipping >2012-11-08 09:50:01,126 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-52) [63bd4906] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:50:01,158 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-52) [63bd4906] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:50:01,161 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-52) [63bd4906] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:50:01,178 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-52) [63bd4906] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 43df365f >2012-11-08 09:50:01,207 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-52) [63bd4906] spmStart polling started: taskId = aa1b8862-ad3b-41ab-bc2e-30a7ba14f49e >2012-11-08 09:50:02,252 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-52) [63bd4906] Failed in HSMGetTaskStatusVDS method >2012-11-08 09:50:02,252 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-52) [63bd4906] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:50:02,252 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-52) [63bd4906] spmStart polling ended: taskId = aa1b8862-ad3b-41ab-bc2e-30a7ba14f49e task status = finished >2012-11-08 09:50:02,252 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-52) [63bd4906] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:50:02,279 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-52) [63bd4906] spmStart polling ended, spm status: Free >2012-11-08 09:50:02,306 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-52) [63bd4906] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=aa1b8862-ad3b-41ab-bc2e-30a7ba14f49e), log id: e5b1e78 >2012-11-08 09:50:02,327 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-52) [63bd4906] FINISH, HSMClearTaskVDSCommand, log id: e5b1e78 >2012-11-08 09:50:02,327 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-52) [63bd4906] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2d5fa5, log id: 43df365f >2012-11-08 09:50:02,329 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-52) [22e249ba] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:50:02,352 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-52) [22e249ba] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:50:12,447 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-89) hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:50:12,480 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-89) SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:50:12,483 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-89) starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:50:12,498 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-89) START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 580ae806 >2012-11-08 09:50:12,527 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-89) spmStart polling started: taskId = d66ce061-4046-482c-9dcf-d350a9e38090 >2012-11-08 09:50:13,565 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-89) Failed in HSMGetTaskStatusVDS method >2012-11-08 09:50:13,565 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-89) Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:50:13,565 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-89) spmStart polling ended: taskId = d66ce061-4046-482c-9dcf-d350a9e38090 task status = finished >2012-11-08 09:50:13,565 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-89) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:50:13,604 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-89) spmStart polling ended, spm status: Free >2012-11-08 09:50:13,614 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-89) START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=d66ce061-4046-482c-9dcf-d350a9e38090), log id: 303329e5 >2012-11-08 09:50:13,635 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-89) FINISH, HSMClearTaskVDSCommand, log id: 303329e5 >2012-11-08 09:50:13,635 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-89) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6f03921b, log id: 580ae806 >2012-11-08 09:50:13,637 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-89) [536d638d] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:50:13,661 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-89) [536d638d] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:50:23,741 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-15) hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:50:23,770 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-15) SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:50:23,773 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-15) starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:50:23,788 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-15) START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7ce57431 >2012-11-08 09:50:23,817 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-15) spmStart polling started: taskId = b8b7d545-ad95-4ce7-bc24-74699802a244 >2012-11-08 09:50:24,854 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-15) Failed in HSMGetTaskStatusVDS method >2012-11-08 09:50:24,854 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-15) Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:50:24,854 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-15) spmStart polling ended: taskId = b8b7d545-ad95-4ce7-bc24-74699802a244 task status = finished >2012-11-08 09:50:24,854 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-15) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:50:24,899 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-15) spmStart polling ended, spm status: Free >2012-11-08 09:50:24,912 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-15) START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=b8b7d545-ad95-4ce7-bc24-74699802a244), log id: 6e96b8e9 >2012-11-08 09:50:24,934 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-15) FINISH, HSMClearTaskVDSCommand, log id: 6e96b8e9 >2012-11-08 09:50:24,934 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-15) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@694601b7, log id: 7ce57431 >2012-11-08 09:50:24,935 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-15) [731acb5f] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:50:24,950 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-15) [731acb5f] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:50:35,009 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [491550a0] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:50:35,039 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [491550a0] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:50:35,042 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [491550a0] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:50:35,055 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [491550a0] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 61ffb35f >2012-11-08 09:50:35,076 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [491550a0] spmStart polling started: taskId = eed43f35-5929-4986-92dc-058a674a07cd >2012-11-08 09:50:36,108 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-94) [491550a0] Failed in HSMGetTaskStatusVDS method >2012-11-08 09:50:36,108 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-94) [491550a0] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:50:36,108 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [491550a0] spmStart polling ended: taskId = eed43f35-5929-4986-92dc-058a674a07cd task status = finished >2012-11-08 09:50:36,108 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [491550a0] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:50:36,136 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [491550a0] spmStart polling ended, spm status: Free >2012-11-08 09:50:36,154 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-94) [491550a0] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=eed43f35-5929-4986-92dc-058a674a07cd), log id: 645deb1b >2012-11-08 09:50:36,171 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-94) [491550a0] FINISH, HSMClearTaskVDSCommand, log id: 645deb1b >2012-11-08 09:50:36,171 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [491550a0] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@51f66d17, log id: 61ffb35f >2012-11-08 09:50:36,173 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-94) [9742b0b] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:50:36,193 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [9742b0b] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:50:46,250 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [7835b95d] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:50:46,276 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [7835b95d] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:50:46,279 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [7835b95d] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:50:46,295 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [7835b95d] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 400bcf72 >2012-11-08 09:50:46,317 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [7835b95d] spmStart polling started: taskId = ff732584-54b0-4430-bf9e-bbb9396fc72d >2012-11-08 09:50:47,349 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-69) [7835b95d] Failed in HSMGetTaskStatusVDS method >2012-11-08 09:50:47,349 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-69) [7835b95d] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:50:47,349 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [7835b95d] spmStart polling ended: taskId = ff732584-54b0-4430-bf9e-bbb9396fc72d task status = finished >2012-11-08 09:50:47,349 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [7835b95d] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:50:47,377 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [7835b95d] spmStart polling ended, spm status: Free >2012-11-08 09:50:47,390 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-69) [7835b95d] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=ff732584-54b0-4430-bf9e-bbb9396fc72d), log id: 1bdd13b0 >2012-11-08 09:50:47,408 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-69) [7835b95d] FINISH, HSMClearTaskVDSCommand, log id: 1bdd13b0 >2012-11-08 09:50:47,408 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [7835b95d] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4f37961a, log id: 400bcf72 >2012-11-08 09:50:47,410 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-69) [4a5fad97] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:50:47,432 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [4a5fad97] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:50:48,804 INFO [org.ovirt.engine.core.bll.SetVmTicketCommand] (ajp-/127.0.0.1:8702-16) [f170980] Running command: SetVmTicketCommand internal: false. Entities affected : ID: 28f91cc7-0841-425d-a0fd-8ea1fd15c636 Type: VM >2012-11-08 09:50:48,829 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp-/127.0.0.1:8702-16) [f170980] START, SetVmTicketVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=28f91cc7-0841-425d-a0fd-8ea1fd15c636, ticket=sA0EAoiYfMQo, validTime=120,m userName=admin@internal, userId=fdfc627c-d875-11e0-90f0-83df133b58cc), log id: 57d28ca >2012-11-08 09:50:48,862 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp-/127.0.0.1:8702-16) [f170980] FINISH, SetVmTicketVDSCommand, log id: 57d28ca >2012-11-08 09:50:57,498 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) [4f2011dc] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:50:57,525 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) [4f2011dc] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:50:57,528 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) [4f2011dc] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:50:57,545 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-51) [4f2011dc] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7e34d9d2 >2012-11-08 09:50:57,570 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-51) [4f2011dc] spmStart polling started: taskId = 098d5760-3fb5-4daa-a06c-5732ad5788ff >2012-11-08 09:50:58,603 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-51) [4f2011dc] Failed in HSMGetTaskStatusVDS method >2012-11-08 09:50:58,603 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-51) [4f2011dc] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:50:58,603 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-51) [4f2011dc] spmStart polling ended: taskId = 098d5760-3fb5-4daa-a06c-5732ad5788ff task status = finished >2012-11-08 09:50:58,603 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-51) [4f2011dc] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:50:58,634 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-51) [4f2011dc] spmStart polling ended, spm status: Free >2012-11-08 09:50:58,653 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-51) [4f2011dc] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=098d5760-3fb5-4daa-a06c-5732ad5788ff), log id: 72540efa >2012-11-08 09:50:58,678 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-51) [4f2011dc] FINISH, HSMClearTaskVDSCommand, log id: 72540efa >2012-11-08 09:50:58,679 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-51) [4f2011dc] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6ca379b9, log id: 7e34d9d2 >2012-11-08 09:50:58,681 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-51) [608c9867] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:50:58,709 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) [608c9867] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:51:08,769 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) [1d711110] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:51:08,795 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) [1d711110] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:51:08,797 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) [1d711110] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:51:08,814 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-60) [1d711110] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 703da13d >2012-11-08 09:51:08,836 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-60) [1d711110] spmStart polling started: taskId = a781edcb-2e53-4b96-9a5e-ebbaa2237745 >2012-11-08 09:51:09,867 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-60) [1d711110] Failed in HSMGetTaskStatusVDS method >2012-11-08 09:51:09,867 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-60) [1d711110] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:51:09,867 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-60) [1d711110] spmStart polling ended: taskId = a781edcb-2e53-4b96-9a5e-ebbaa2237745 task status = finished >2012-11-08 09:51:09,867 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-60) [1d711110] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:51:09,895 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-60) [1d711110] spmStart polling ended, spm status: Free >2012-11-08 09:51:09,908 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-60) [1d711110] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=a781edcb-2e53-4b96-9a5e-ebbaa2237745), log id: 697b764a >2012-11-08 09:51:09,926 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-60) [1d711110] FINISH, HSMClearTaskVDSCommand, log id: 697b764a >2012-11-08 09:51:09,926 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-60) [1d711110] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@28932a60, log id: 703da13d >2012-11-08 09:51:09,928 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-60) [55c3196f] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:51:09,947 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) [55c3196f] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:51:18,787 INFO [org.ovirt.engine.core.bll.SetVmTicketCommand] (ajp-/127.0.0.1:8702-4) [24147573] Running command: SetVmTicketCommand internal: false. Entities affected : ID: 0ed98fdf-e2d5-4bf6-b94a-3cba410c4c8f Type: VM >2012-11-08 09:51:18,807 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp-/127.0.0.1:8702-4) [24147573] START, SetVmTicketVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=0ed98fdf-e2d5-4bf6-b94a-3cba410c4c8f, ticket=84tMqIUTK59w, validTime=120,m userName=admin@internal, userId=fdfc627c-d875-11e0-90f0-83df133b58cc), log id: 731576af >2012-11-08 09:51:18,872 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp-/127.0.0.1:8702-4) [24147573] FINISH, SetVmTicketVDSCommand, log id: 731576af >2012-11-08 09:51:19,995 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [4e4f0a76] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:51:20,024 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [4e4f0a76] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:51:20,026 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [4e4f0a76] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:51:20,043 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-98) [4e4f0a76] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 343b785b >2012-11-08 09:51:20,065 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-98) [4e4f0a76] spmStart polling started: taskId = 4e546153-c40c-4dee-b138-6e8e55ffa132 >2012-11-08 09:51:21,094 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-98) [4e4f0a76] Failed in HSMGetTaskStatusVDS method >2012-11-08 09:51:21,094 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-98) [4e4f0a76] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:51:21,094 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-98) [4e4f0a76] spmStart polling ended: taskId = 4e546153-c40c-4dee-b138-6e8e55ffa132 task status = finished >2012-11-08 09:51:21,094 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-98) [4e4f0a76] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:51:21,124 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-98) [4e4f0a76] spmStart polling ended, spm status: Free >2012-11-08 09:51:21,134 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-98) [4e4f0a76] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=4e546153-c40c-4dee-b138-6e8e55ffa132), log id: 7f833bdc >2012-11-08 09:51:21,159 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-98) [4e4f0a76] FINISH, HSMClearTaskVDSCommand, log id: 7f833bdc >2012-11-08 09:51:21,159 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-98) [4e4f0a76] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@cf29eb, log id: 343b785b >2012-11-08 09:51:21,160 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-98) [20c903b7] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:51:21,178 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [20c903b7] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:51:31,235 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-24) [3f16a62d] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:51:31,263 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-24) [3f16a62d] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:51:31,266 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-24) [3f16a62d] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:51:31,283 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-24) [3f16a62d] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 33cb8e4b >2012-11-08 09:51:31,312 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-24) [3f16a62d] spmStart polling started: taskId = 4e051cd5-cf8f-4d65-b279-94224479d1b0 >2012-11-08 09:51:32,350 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-24) [3f16a62d] Failed in HSMGetTaskStatusVDS method >2012-11-08 09:51:32,350 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-24) [3f16a62d] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:51:32,350 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-24) [3f16a62d] spmStart polling ended: taskId = 4e051cd5-cf8f-4d65-b279-94224479d1b0 task status = finished >2012-11-08 09:51:32,350 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-24) [3f16a62d] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:51:32,377 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-24) [3f16a62d] spmStart polling ended, spm status: Free >2012-11-08 09:51:32,389 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-24) [3f16a62d] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=4e051cd5-cf8f-4d65-b279-94224479d1b0), log id: 7a7e9b6e >2012-11-08 09:51:32,411 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-24) [3f16a62d] FINISH, HSMClearTaskVDSCommand, log id: 7a7e9b6e >2012-11-08 09:51:32,411 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-24) [3f16a62d] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@e0e9196, log id: 33cb8e4b >2012-11-08 09:51:32,412 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-24) [5e78160f] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:51:32,429 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-24) [5e78160f] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:51:42,486 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-52) [22e249ba] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:51:42,513 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-52) [22e249ba] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:51:42,516 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-52) [22e249ba] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:51:42,533 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-52) [22e249ba] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5f70d751 >2012-11-08 09:51:42,562 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-52) [22e249ba] spmStart polling started: taskId = a0554462-1bdd-4e92-af24-350635f9e176 >2012-11-08 09:51:43,607 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-52) [22e249ba] Failed in HSMGetTaskStatusVDS method >2012-11-08 09:51:43,607 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-52) [22e249ba] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:51:43,607 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-52) [22e249ba] spmStart polling ended: taskId = a0554462-1bdd-4e92-af24-350635f9e176 task status = finished >2012-11-08 09:51:43,607 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-52) [22e249ba] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:51:43,635 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-52) [22e249ba] spmStart polling ended, spm status: Free >2012-11-08 09:51:43,647 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-52) [22e249ba] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=a0554462-1bdd-4e92-af24-350635f9e176), log id: 67683167 >2012-11-08 09:51:43,669 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-52) [22e249ba] FINISH, HSMClearTaskVDSCommand, log id: 67683167 >2012-11-08 09:51:43,669 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-52) [22e249ba] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6983963a, log id: 5f70d751 >2012-11-08 09:51:43,671 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-52) [3dfbeff4] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:51:43,697 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-52) [3dfbeff4] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:51:53,759 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-89) [536d638d] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:51:53,786 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-89) [536d638d] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:51:53,788 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-89) [536d638d] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:51:53,799 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-89) [536d638d] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 44a009e2 >2012-11-08 09:51:53,825 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-89) [536d638d] spmStart polling started: taskId = b8af21f6-8db5-4b77-8f57-6cdc01786349 >2012-11-08 09:51:54,867 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-89) [536d638d] Failed in HSMGetTaskStatusVDS method >2012-11-08 09:51:54,867 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-89) [536d638d] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:51:54,867 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-89) [536d638d] spmStart polling ended: taskId = b8af21f6-8db5-4b77-8f57-6cdc01786349 task status = finished >2012-11-08 09:51:54,867 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-89) [536d638d] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:51:54,894 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-89) [536d638d] spmStart polling ended, spm status: Free >2012-11-08 09:51:54,907 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-89) [536d638d] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=b8af21f6-8db5-4b77-8f57-6cdc01786349), log id: d8455bc >2012-11-08 09:51:54,930 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-89) [536d638d] FINISH, HSMClearTaskVDSCommand, log id: d8455bc >2012-11-08 09:51:54,930 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-89) [536d638d] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@78d1934d, log id: 44a009e2 >2012-11-08 09:51:54,931 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-89) [71de233] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:51:54,948 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-89) [71de233] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:52:05,001 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) [608c9867] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:52:05,031 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) [608c9867] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:52:05,034 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) [608c9867] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:52:05,051 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-51) [608c9867] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5ae8ce42 >2012-11-08 09:52:05,074 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-51) [608c9867] spmStart polling started: taskId = 5433fbae-7bba-4419-98e3-8574f445b774 >2012-11-08 09:52:06,111 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-51) [608c9867] Failed in HSMGetTaskStatusVDS method >2012-11-08 09:52:06,111 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-51) [608c9867] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:52:06,112 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-51) [608c9867] spmStart polling ended: taskId = 5433fbae-7bba-4419-98e3-8574f445b774 task status = finished >2012-11-08 09:52:06,112 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-51) [608c9867] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:52:06,142 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-51) [608c9867] spmStart polling ended, spm status: Free >2012-11-08 09:52:06,160 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-51) [608c9867] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=5433fbae-7bba-4419-98e3-8574f445b774), log id: 7cd5f5fa >2012-11-08 09:52:06,179 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-51) [608c9867] FINISH, HSMClearTaskVDSCommand, log id: 7cd5f5fa >2012-11-08 09:52:06,179 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-51) [608c9867] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4308301d, log id: 5ae8ce42 >2012-11-08 09:52:06,181 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-51) [4e8ee24c] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:52:06,205 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) [4e8ee24c] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:52:16,271 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [9742b0b] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:52:16,300 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [9742b0b] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:52:16,303 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [9742b0b] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:52:16,321 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [9742b0b] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 734faa08 >2012-11-08 09:52:16,351 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [9742b0b] spmStart polling started: taskId = 36467803-b880-482f-b872-0a2d15a08da3 >2012-11-08 09:52:17,388 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-94) [9742b0b] Failed in HSMGetTaskStatusVDS method >2012-11-08 09:52:17,388 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-94) [9742b0b] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:52:17,388 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [9742b0b] spmStart polling ended: taskId = 36467803-b880-482f-b872-0a2d15a08da3 task status = finished >2012-11-08 09:52:17,388 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [9742b0b] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:52:17,420 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [9742b0b] spmStart polling ended, spm status: Free >2012-11-08 09:52:17,434 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-94) [9742b0b] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=36467803-b880-482f-b872-0a2d15a08da3), log id: 5c8da93c >2012-11-08 09:52:17,457 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-94) [9742b0b] FINISH, HSMClearTaskVDSCommand, log id: 5c8da93c >2012-11-08 09:52:17,457 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [9742b0b] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@5861d19c, log id: 734faa08 >2012-11-08 09:52:17,459 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-94) [7e088618] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:52:17,475 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [7e088618] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:52:27,532 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-89) [71de233] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:52:27,559 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-89) [71de233] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:52:27,562 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-89) [71de233] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:52:27,577 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-89) [71de233] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3510f2ea >2012-11-08 09:52:27,603 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-89) [71de233] spmStart polling started: taskId = 7fe77e80-94b4-4a36-906d-f5a4ba177059 >2012-11-08 09:52:28,641 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-89) [71de233] Failed in HSMGetTaskStatusVDS method >2012-11-08 09:52:28,641 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-89) [71de233] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:52:28,641 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-89) [71de233] spmStart polling ended: taskId = 7fe77e80-94b4-4a36-906d-f5a4ba177059 task status = finished >2012-11-08 09:52:28,642 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-89) [71de233] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:52:28,670 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-89) [71de233] spmStart polling ended, spm status: Free >2012-11-08 09:52:28,684 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-89) [71de233] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=7fe77e80-94b4-4a36-906d-f5a4ba177059), log id: 7a89bb37 >2012-11-08 09:52:28,707 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-89) [71de233] FINISH, HSMClearTaskVDSCommand, log id: 7a89bb37 >2012-11-08 09:52:28,707 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-89) [71de233] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@686256ba, log id: 3510f2ea >2012-11-08 09:52:28,709 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-89) [102c02e0] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:52:28,725 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-89) [102c02e0] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:52:38,786 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-15) [731acb5f] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:52:38,812 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-15) [731acb5f] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:52:38,815 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-15) [731acb5f] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:52:38,831 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-15) [731acb5f] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 29bbb29 >2012-11-08 09:52:38,853 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-15) [731acb5f] spmStart polling started: taskId = 8c96ef1f-26b1-4b1e-9f93-a7fff16dc3e3 >2012-11-08 09:52:39,888 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-15) [731acb5f] Failed in HSMGetTaskStatusVDS method >2012-11-08 09:52:39,888 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-15) [731acb5f] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:52:39,888 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-15) [731acb5f] spmStart polling ended: taskId = 8c96ef1f-26b1-4b1e-9f93-a7fff16dc3e3 task status = finished >2012-11-08 09:52:39,888 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-15) [731acb5f] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:52:39,919 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-15) [731acb5f] spmStart polling ended, spm status: Free >2012-11-08 09:52:39,938 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-15) [731acb5f] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=8c96ef1f-26b1-4b1e-9f93-a7fff16dc3e3), log id: 7339034c >2012-11-08 09:52:39,959 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-15) [731acb5f] FINISH, HSMClearTaskVDSCommand, log id: 7339034c >2012-11-08 09:52:39,959 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-15) [731acb5f] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2407adad, log id: 29bbb29 >2012-11-08 09:52:39,961 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-15) [741b4b55] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:52:39,985 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-15) [741b4b55] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:52:50,050 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [57e8ea4b] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:52:50,079 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [57e8ea4b] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:52:50,081 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [57e8ea4b] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:52:50,099 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [57e8ea4b] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4f2c11bc >2012-11-08 09:52:50,124 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [57e8ea4b] spmStart polling started: taskId = 37279036-85b8-449e-a97f-f16437efa9ec >2012-11-08 09:52:51,165 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-72) [57e8ea4b] Failed in HSMGetTaskStatusVDS method >2012-11-08 09:52:51,165 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-72) [57e8ea4b] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:52:51,165 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [57e8ea4b] spmStart polling ended: taskId = 37279036-85b8-449e-a97f-f16437efa9ec task status = finished >2012-11-08 09:52:51,165 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [57e8ea4b] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:52:51,191 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [57e8ea4b] spmStart polling ended, spm status: Free >2012-11-08 09:52:51,204 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-72) [57e8ea4b] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=37279036-85b8-449e-a97f-f16437efa9ec), log id: 3f69595 >2012-11-08 09:52:51,226 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-72) [57e8ea4b] FINISH, HSMClearTaskVDSCommand, log id: 3f69595 >2012-11-08 09:52:51,226 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [57e8ea4b] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@31d66b3b, log id: 4f2c11bc >2012-11-08 09:52:51,228 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-72) [741a080c] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:52:51,253 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [741a080c] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:53:01,318 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-89) [102c02e0] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:53:01,343 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-89) [102c02e0] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:53:01,345 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-89) [102c02e0] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:53:01,361 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-89) [102c02e0] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 50116a44 >2012-11-08 09:53:01,386 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-89) [102c02e0] spmStart polling started: taskId = fd722308-29ed-470f-b1f0-dd45b28621d9 >2012-11-08 09:53:02,425 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-89) [102c02e0] Failed in HSMGetTaskStatusVDS method >2012-11-08 09:53:02,425 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-89) [102c02e0] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:53:02,425 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-89) [102c02e0] spmStart polling ended: taskId = fd722308-29ed-470f-b1f0-dd45b28621d9 task status = finished >2012-11-08 09:53:02,425 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-89) [102c02e0] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:53:02,452 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-89) [102c02e0] spmStart polling ended, spm status: Free >2012-11-08 09:53:02,464 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-89) [102c02e0] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=fd722308-29ed-470f-b1f0-dd45b28621d9), log id: 7f1244f2 >2012-11-08 09:53:02,487 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-89) [102c02e0] FINISH, HSMClearTaskVDSCommand, log id: 7f1244f2 >2012-11-08 09:53:02,487 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-89) [102c02e0] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@86cad0b, log id: 50116a44 >2012-11-08 09:53:02,488 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-89) [16d4b17f] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:53:02,512 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-89) [16d4b17f] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:53:12,584 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-26) [733c0daa] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:53:12,611 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-26) [733c0daa] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:53:12,614 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-26) [733c0daa] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:53:12,628 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-26) [733c0daa] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6607f399 >2012-11-08 09:53:12,649 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-26) [733c0daa] spmStart polling started: taskId = 0e28cf8e-e671-4327-a269-527f32890a4a >2012-11-08 09:53:13,687 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-26) [733c0daa] Failed in HSMGetTaskStatusVDS method >2012-11-08 09:53:13,687 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-26) [733c0daa] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:53:13,687 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-26) [733c0daa] spmStart polling ended: taskId = 0e28cf8e-e671-4327-a269-527f32890a4a task status = finished >2012-11-08 09:53:13,688 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-26) [733c0daa] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:53:13,719 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-26) [733c0daa] spmStart polling ended, spm status: Free >2012-11-08 09:53:13,737 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-26) [733c0daa] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=0e28cf8e-e671-4327-a269-527f32890a4a), log id: 6de3f355 >2012-11-08 09:53:13,755 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-26) [733c0daa] FINISH, HSMClearTaskVDSCommand, log id: 6de3f355 >2012-11-08 09:53:13,755 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-26) [733c0daa] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@107c89d0, log id: 6607f399 >2012-11-08 09:53:13,757 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-26) [603bbf24] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:53:13,782 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-26) [603bbf24] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:53:23,835 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [264a996b] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:53:23,859 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [264a996b] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:53:23,862 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [264a996b] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:53:23,873 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-59) [264a996b] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7c45235d >2012-11-08 09:53:23,904 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-59) [264a996b] spmStart polling started: taskId = 45fd931f-952d-41c4-8b4e-331ff3defaa9 >2012-11-08 09:53:24,938 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-59) [264a996b] Failed in HSMGetTaskStatusVDS method >2012-11-08 09:53:24,938 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-59) [264a996b] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:53:24,938 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-59) [264a996b] spmStart polling ended: taskId = 45fd931f-952d-41c4-8b4e-331ff3defaa9 task status = finished >2012-11-08 09:53:24,938 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-59) [264a996b] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:53:24,967 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-59) [264a996b] spmStart polling ended, spm status: Free >2012-11-08 09:53:24,986 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-59) [264a996b] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=45fd931f-952d-41c4-8b4e-331ff3defaa9), log id: 3ddd0a32 >2012-11-08 09:53:25,008 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-59) [264a996b] FINISH, HSMClearTaskVDSCommand, log id: 3ddd0a32 >2012-11-08 09:53:25,008 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-59) [264a996b] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2877ad44, log id: 7c45235d >2012-11-08 09:53:25,009 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-59) [300b6256] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:53:25,026 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-59) [300b6256] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:53:35,081 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-100) hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:53:35,108 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-100) SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:53:35,111 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-100) starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:53:35,128 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-100) START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 68e02f11 >2012-11-08 09:53:35,150 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-100) spmStart polling started: taskId = 4ae1854e-e5bc-45e3-be00-0debfa59bff8 >2012-11-08 09:53:36,180 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-100) Failed in HSMGetTaskStatusVDS method >2012-11-08 09:53:36,180 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-100) Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:53:36,180 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-100) spmStart polling ended: taskId = 4ae1854e-e5bc-45e3-be00-0debfa59bff8 task status = finished >2012-11-08 09:53:36,181 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-100) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:53:36,207 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-100) spmStart polling ended, spm status: Free >2012-11-08 09:53:36,221 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-100) START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=4ae1854e-e5bc-45e3-be00-0debfa59bff8), log id: 50d70903 >2012-11-08 09:53:36,238 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-100) FINISH, HSMClearTaskVDSCommand, log id: 50d70903 >2012-11-08 09:53:36,238 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-100) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@5590cd6b, log id: 68e02f11 >2012-11-08 09:53:36,240 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-100) [159e57b0] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:53:36,259 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-100) [159e57b0] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:53:46,316 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-24) [5e78160f] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:53:46,345 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-24) [5e78160f] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:53:46,348 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-24) [5e78160f] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:53:46,366 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-24) [5e78160f] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3e897173 >2012-11-08 09:53:46,395 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-24) [5e78160f] spmStart polling started: taskId = 385846cb-2de4-4c23-8cb2-1036174f432f >2012-11-08 09:53:47,429 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-24) [5e78160f] Failed in HSMGetTaskStatusVDS method >2012-11-08 09:53:47,429 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-24) [5e78160f] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:53:47,429 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-24) [5e78160f] spmStart polling ended: taskId = 385846cb-2de4-4c23-8cb2-1036174f432f task status = finished >2012-11-08 09:53:47,429 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-24) [5e78160f] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:53:47,458 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-24) [5e78160f] spmStart polling ended, spm status: Free >2012-11-08 09:53:47,471 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-24) [5e78160f] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=385846cb-2de4-4c23-8cb2-1036174f432f), log id: 6b24cc55 >2012-11-08 09:53:47,494 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-24) [5e78160f] FINISH, HSMClearTaskVDSCommand, log id: 6b24cc55 >2012-11-08 09:53:47,494 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-24) [5e78160f] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3ce31767, log id: 3e897173 >2012-11-08 09:53:47,495 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-24) [61158a6b] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:53:47,513 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-24) [61158a6b] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:53:57,570 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [741a080c] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:53:57,597 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [741a080c] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:53:57,599 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [741a080c] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:53:57,617 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [741a080c] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3eb7e8a7 >2012-11-08 09:53:57,640 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [741a080c] spmStart polling started: taskId = cfa4904c-74c2-479c-98fc-69cfbb9c6281 >2012-11-08 09:53:58,670 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-72) [741a080c] Failed in HSMGetTaskStatusVDS method >2012-11-08 09:53:58,671 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-72) [741a080c] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:53:58,671 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [741a080c] spmStart polling ended: taskId = cfa4904c-74c2-479c-98fc-69cfbb9c6281 task status = finished >2012-11-08 09:53:58,671 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [741a080c] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:53:58,699 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [741a080c] spmStart polling ended, spm status: Free >2012-11-08 09:53:58,716 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-72) [741a080c] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=cfa4904c-74c2-479c-98fc-69cfbb9c6281), log id: 36e1ce8d >2012-11-08 09:53:58,733 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-72) [741a080c] FINISH, HSMClearTaskVDSCommand, log id: 36e1ce8d >2012-11-08 09:53:58,733 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [741a080c] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@22bfa721, log id: 3eb7e8a7 >2012-11-08 09:53:58,735 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-72) [54757959] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:53:58,755 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [54757959] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:54:08,810 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [4378e66c] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:54:08,831 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [4378e66c] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:54:08,833 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [4378e66c] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:54:08,843 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-87) [4378e66c] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 1feb4e3f >2012-11-08 09:54:08,869 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-87) [4378e66c] spmStart polling started: taskId = 8e047b72-1f89-4e44-90e0-68901e2b19b5 >2012-11-08 09:54:09,905 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-87) [4378e66c] Failed in HSMGetTaskStatusVDS method >2012-11-08 09:54:09,905 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-87) [4378e66c] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:54:09,905 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-87) [4378e66c] spmStart polling ended: taskId = 8e047b72-1f89-4e44-90e0-68901e2b19b5 task status = finished >2012-11-08 09:54:09,905 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-87) [4378e66c] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:54:09,936 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-87) [4378e66c] spmStart polling ended, spm status: Free >2012-11-08 09:54:09,949 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-87) [4378e66c] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=8e047b72-1f89-4e44-90e0-68901e2b19b5), log id: 63fc0ae1 >2012-11-08 09:54:09,973 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-87) [4378e66c] FINISH, HSMClearTaskVDSCommand, log id: 63fc0ae1 >2012-11-08 09:54:09,973 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-87) [4378e66c] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@382e6d0f, log id: 1feb4e3f >2012-11-08 09:54:09,974 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-87) [6ffa1099] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:54:09,990 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-87) [6ffa1099] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:54:20,051 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-15) [741b4b55] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:54:20,077 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-15) [741b4b55] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:54:20,080 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-15) [741b4b55] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:54:20,097 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-15) [741b4b55] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: d06b7fa >2012-11-08 09:54:20,126 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-15) [741b4b55] spmStart polling started: taskId = 528fa262-7f2d-49d8-9b9b-d5dcab078fe5 >2012-11-08 09:54:21,170 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-15) [741b4b55] Failed in HSMGetTaskStatusVDS method >2012-11-08 09:54:21,170 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-15) [741b4b55] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:54:21,170 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-15) [741b4b55] spmStart polling ended: taskId = 528fa262-7f2d-49d8-9b9b-d5dcab078fe5 task status = finished >2012-11-08 09:54:21,170 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-15) [741b4b55] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:54:21,198 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-15) [741b4b55] spmStart polling ended, spm status: Free >2012-11-08 09:54:21,210 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-15) [741b4b55] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=528fa262-7f2d-49d8-9b9b-d5dcab078fe5), log id: 6731e349 >2012-11-08 09:54:21,232 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-15) [741b4b55] FINISH, HSMClearTaskVDSCommand, log id: 6731e349 >2012-11-08 09:54:21,232 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-15) [741b4b55] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7c41028c, log id: d06b7fa >2012-11-08 09:54:21,234 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-15) [100e750f] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:54:21,253 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-15) [100e750f] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:54:31,320 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-56) [4783ef00] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:54:31,351 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-56) [4783ef00] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:54:31,354 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-56) [4783ef00] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:54:31,368 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-56) [4783ef00] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 52becc67 >2012-11-08 09:54:31,391 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-56) [4783ef00] spmStart polling started: taskId = 417ab127-ec7e-47fb-be76-747f04951d2b >2012-11-08 09:54:32,427 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-56) [4783ef00] Failed in HSMGetTaskStatusVDS method >2012-11-08 09:54:32,427 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-56) [4783ef00] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:54:32,428 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-56) [4783ef00] spmStart polling ended: taskId = 417ab127-ec7e-47fb-be76-747f04951d2b task status = finished >2012-11-08 09:54:32,428 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-56) [4783ef00] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:54:32,457 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-56) [4783ef00] spmStart polling ended, spm status: Free >2012-11-08 09:54:32,469 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-56) [4783ef00] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=417ab127-ec7e-47fb-be76-747f04951d2b), log id: 1f40d078 >2012-11-08 09:54:32,493 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-56) [4783ef00] FINISH, HSMClearTaskVDSCommand, log id: 1f40d078 >2012-11-08 09:54:32,493 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-56) [4783ef00] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@5836864f, log id: 52becc67 >2012-11-08 09:54:32,495 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-56) [63abd9c7] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:54:32,520 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-56) [63abd9c7] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:54:34,205 INFO [org.ovirt.engine.core.bll.SetVmTicketCommand] (ajp-/127.0.0.1:8702-14) [18948d7b] Running command: SetVmTicketCommand internal: false. Entities affected : ID: 8a0d2e26-440c-4d6c-91d3-1fc4d9b75990 Type: VM >2012-11-08 09:54:34,236 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp-/127.0.0.1:8702-14) [18948d7b] START, SetVmTicketVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=8a0d2e26-440c-4d6c-91d3-1fc4d9b75990, ticket=raRAzTn3IrAG, validTime=120,m userName=admin@internal, userId=fdfc627c-d875-11e0-90f0-83df133b58cc), log id: 2afa9bda >2012-11-08 09:54:34,265 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp-/127.0.0.1:8702-14) [18948d7b] FINISH, SetVmTicketVDSCommand, log id: 2afa9bda >2012-11-08 09:54:42,590 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-100) [159e57b0] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:54:42,616 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-100) [159e57b0] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:54:42,619 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-100) [159e57b0] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:54:42,635 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-100) [159e57b0] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 527950e1 >2012-11-08 09:54:42,659 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-100) [159e57b0] spmStart polling started: taskId = b8643544-946d-4eb1-9463-b1b5d9af7f64 >2012-11-08 09:54:43,699 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-100) [159e57b0] Failed in HSMGetTaskStatusVDS method >2012-11-08 09:54:43,700 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-100) [159e57b0] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:54:43,700 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-100) [159e57b0] spmStart polling ended: taskId = b8643544-946d-4eb1-9463-b1b5d9af7f64 task status = finished >2012-11-08 09:54:43,700 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-100) [159e57b0] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:54:43,725 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-100) [159e57b0] spmStart polling ended, spm status: Free >2012-11-08 09:54:43,744 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-100) [159e57b0] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=b8643544-946d-4eb1-9463-b1b5d9af7f64), log id: 646e95ea >2012-11-08 09:54:43,764 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-100) [159e57b0] FINISH, HSMClearTaskVDSCommand, log id: 646e95ea >2012-11-08 09:54:43,764 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-100) [159e57b0] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@18346929, log id: 527950e1 >2012-11-08 09:54:43,766 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-100) [18e40f29] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:54:43,789 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-100) [18e40f29] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:54:53,853 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-26) [603bbf24] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:54:53,878 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-26) [603bbf24] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:54:53,880 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-26) [603bbf24] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:54:53,896 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-26) [603bbf24] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 13171134 >2012-11-08 09:54:53,919 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-26) [603bbf24] spmStart polling started: taskId = 4e84b952-efb7-407a-b1fe-a052038c4bd9 >2012-11-08 09:54:54,953 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-26) [603bbf24] Failed in HSMGetTaskStatusVDS method >2012-11-08 09:54:54,953 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-26) [603bbf24] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:54:54,953 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-26) [603bbf24] spmStart polling ended: taskId = 4e84b952-efb7-407a-b1fe-a052038c4bd9 task status = finished >2012-11-08 09:54:54,953 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-26) [603bbf24] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:54:54,986 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-26) [603bbf24] spmStart polling ended, spm status: Free >2012-11-08 09:54:54,998 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-26) [603bbf24] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=4e84b952-efb7-407a-b1fe-a052038c4bd9), log id: 51386fc6 >2012-11-08 09:54:55,021 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-26) [603bbf24] FINISH, HSMClearTaskVDSCommand, log id: 51386fc6 >2012-11-08 09:54:55,021 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-26) [603bbf24] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@503259d2, log id: 13171134 >2012-11-08 09:54:55,023 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-26) [19a71320] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:54:55,047 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-26) [19a71320] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:55:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-29) [251cffc8] Autorecovering hosts is disabled, skipping >2012-11-08 09:55:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-29) [251cffc8] Autorecovering storage domains is disabled, skipping >2012-11-08 09:55:05,109 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [16a90de3] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:55:05,142 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [16a90de3] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:55:05,144 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [16a90de3] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:55:05,162 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) [16a90de3] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2e7dfa78 >2012-11-08 09:55:05,189 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) [16a90de3] spmStart polling started: taskId = 637a6ba6-6616-46d7-9124-12db7db7796f >2012-11-08 09:55:05,524 INFO [org.ovirt.engine.core.bll.SetVmTicketCommand] (ajp-/127.0.0.1:8702-11) [875c9d7] Running command: SetVmTicketCommand internal: false. Entities affected : ID: 661d6af9-9024-4744-91b1-258e38466815 Type: VM >2012-11-08 09:55:05,549 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp-/127.0.0.1:8702-11) [875c9d7] START, SetVmTicketVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=661d6af9-9024-4744-91b1-258e38466815, ticket=CuPuKh9SXbDY, validTime=120,m userName=admin@internal, userId=fdfc627c-d875-11e0-90f0-83df133b58cc), log id: 5a60ebcd >2012-11-08 09:55:05,583 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp-/127.0.0.1:8702-11) [875c9d7] FINISH, SetVmTicketVDSCommand, log id: 5a60ebcd >2012-11-08 09:55:06,217 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-61) [16a90de3] Failed in HSMGetTaskStatusVDS method >2012-11-08 09:55:06,218 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-61) [16a90de3] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:55:06,218 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) [16a90de3] spmStart polling ended: taskId = 637a6ba6-6616-46d7-9124-12db7db7796f task status = finished >2012-11-08 09:55:06,218 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) [16a90de3] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:55:06,246 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) [16a90de3] spmStart polling ended, spm status: Free >2012-11-08 09:55:06,262 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-61) [16a90de3] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=637a6ba6-6616-46d7-9124-12db7db7796f), log id: 96482cb >2012-11-08 09:55:06,286 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-61) [16a90de3] FINISH, HSMClearTaskVDSCommand, log id: 96482cb >2012-11-08 09:55:06,286 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) [16a90de3] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@46282d59, log id: 2e7dfa78 >2012-11-08 09:55:06,289 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-61) [77b9520a] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:55:06,310 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [77b9520a] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:55:16,368 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-3) hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:55:16,395 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-3) SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:55:16,397 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-3) starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:55:16,414 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-3) START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3dca8ada >2012-11-08 09:55:16,438 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-3) spmStart polling started: taskId = 052f4b74-96be-481f-bd15-bae66edc9713 >2012-11-08 09:55:17,479 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-3) Failed in HSMGetTaskStatusVDS method >2012-11-08 09:55:17,479 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-3) Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:55:17,479 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-3) spmStart polling ended: taskId = 052f4b74-96be-481f-bd15-bae66edc9713 task status = finished >2012-11-08 09:55:17,479 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-3) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:55:17,521 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-3) spmStart polling ended, spm status: Free >2012-11-08 09:55:17,535 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-3) START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=052f4b74-96be-481f-bd15-bae66edc9713), log id: 127b9b51 >2012-11-08 09:55:17,559 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-3) FINISH, HSMClearTaskVDSCommand, log id: 127b9b51 >2012-11-08 09:55:17,559 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-3) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2110a340, log id: 3dca8ada >2012-11-08 09:55:17,561 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-3) [4b4dcfb5] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:55:17,578 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-3) [4b4dcfb5] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:55:27,650 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-26) [19a71320] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:55:27,676 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-26) [19a71320] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:55:27,679 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-26) [19a71320] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:55:27,694 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-26) [19a71320] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 51fecf13 >2012-11-08 09:55:27,717 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-26) [19a71320] spmStart polling started: taskId = ae44a793-c8c4-4428-ab09-161071ce6994 >2012-11-08 09:55:28,756 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-26) [19a71320] Failed in HSMGetTaskStatusVDS method >2012-11-08 09:55:28,756 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-26) [19a71320] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:55:28,756 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-26) [19a71320] spmStart polling ended: taskId = ae44a793-c8c4-4428-ab09-161071ce6994 task status = finished >2012-11-08 09:55:28,756 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-26) [19a71320] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:55:28,779 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-26) [19a71320] spmStart polling ended, spm status: Free >2012-11-08 09:55:28,789 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-26) [19a71320] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=ae44a793-c8c4-4428-ab09-161071ce6994), log id: 71f166fc >2012-11-08 09:55:28,812 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-26) [19a71320] FINISH, HSMClearTaskVDSCommand, log id: 71f166fc >2012-11-08 09:55:28,812 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-26) [19a71320] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1edd4232, log id: 51fecf13 >2012-11-08 09:55:28,814 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-26) [5024f072] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:55:28,830 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-26) [5024f072] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:55:34,228 INFO [org.ovirt.engine.core.bll.SetVmTicketCommand] (ajp-/127.0.0.1:8702-9) [535e45d0] Running command: SetVmTicketCommand internal: false. Entities affected : ID: 3d766d10-58ab-4306-b5c4-7d109c7b6c46 Type: VM >2012-11-08 09:55:34,262 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp-/127.0.0.1:8702-9) [535e45d0] START, SetVmTicketVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=3d766d10-58ab-4306-b5c4-7d109c7b6c46, ticket=TWO7vti3PiH4, validTime=120,m userName=admin@internal, userId=fdfc627c-d875-11e0-90f0-83df133b58cc), log id: 63951a75 >2012-11-08 09:55:34,302 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp-/127.0.0.1:8702-9) [535e45d0] FINISH, SetVmTicketVDSCommand, log id: 63951a75 >2012-11-08 09:55:38,893 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [54757959] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:55:38,916 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [54757959] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:55:38,919 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [54757959] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:55:38,934 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [54757959] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 10d35c5d >2012-11-08 09:55:38,968 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [54757959] spmStart polling started: taskId = bc1909c8-2daa-40cd-9ef3-d47d3e6c1329 >2012-11-08 09:55:40,005 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-72) [54757959] Failed in HSMGetTaskStatusVDS method >2012-11-08 09:55:40,006 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-72) [54757959] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:55:40,006 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [54757959] spmStart polling ended: taskId = bc1909c8-2daa-40cd-9ef3-d47d3e6c1329 task status = finished >2012-11-08 09:55:40,006 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [54757959] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:55:40,034 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [54757959] spmStart polling ended, spm status: Free >2012-11-08 09:55:40,046 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-72) [54757959] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=bc1909c8-2daa-40cd-9ef3-d47d3e6c1329), log id: 585aef0f >2012-11-08 09:55:40,069 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-72) [54757959] FINISH, HSMClearTaskVDSCommand, log id: 585aef0f >2012-11-08 09:55:40,069 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [54757959] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3a04ef1a, log id: 10d35c5d >2012-11-08 09:55:40,070 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-72) [19edb1b8] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:55:40,088 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [19edb1b8] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:55:50,174 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-89) [16d4b17f] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:55:50,204 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-89) [16d4b17f] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:55:50,206 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-89) [16d4b17f] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:55:50,222 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-89) [16d4b17f] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 573189dd >2012-11-08 09:55:50,246 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-89) [16d4b17f] spmStart polling started: taskId = 28cbe67c-f309-46e0-8752-adb9ea213bc5 >2012-11-08 09:55:51,277 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-89) [16d4b17f] Failed in HSMGetTaskStatusVDS method >2012-11-08 09:55:51,277 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-89) [16d4b17f] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:55:51,277 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-89) [16d4b17f] spmStart polling ended: taskId = 28cbe67c-f309-46e0-8752-adb9ea213bc5 task status = finished >2012-11-08 09:55:51,277 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-89) [16d4b17f] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:55:51,311 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-89) [16d4b17f] spmStart polling ended, spm status: Free >2012-11-08 09:55:51,324 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-89) [16d4b17f] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=28cbe67c-f309-46e0-8752-adb9ea213bc5), log id: 9224f1e >2012-11-08 09:55:51,342 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-89) [16d4b17f] FINISH, HSMClearTaskVDSCommand, log id: 9224f1e >2012-11-08 09:55:51,342 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-89) [16d4b17f] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2b783f41, log id: 573189dd >2012-11-08 09:55:51,344 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-89) [6380dab2] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:55:51,362 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-89) [6380dab2] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:56:01,629 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) [6ce786cd] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:56:01,666 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) [6ce786cd] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:56:01,669 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) [6ce786cd] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:56:01,689 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-31) [6ce786cd] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6994fb74 >2012-11-08 09:56:01,719 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-31) [6ce786cd] spmStart polling started: taskId = d14de0b1-579e-4df5-8407-88c65a945b29 >2012-11-08 09:56:02,757 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-31) [6ce786cd] Failed in HSMGetTaskStatusVDS method >2012-11-08 09:56:02,757 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-31) [6ce786cd] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:56:02,757 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-31) [6ce786cd] spmStart polling ended: taskId = d14de0b1-579e-4df5-8407-88c65a945b29 task status = finished >2012-11-08 09:56:02,757 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-31) [6ce786cd] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:56:02,785 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-31) [6ce786cd] spmStart polling ended, spm status: Free >2012-11-08 09:56:02,801 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-31) [6ce786cd] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=d14de0b1-579e-4df5-8407-88c65a945b29), log id: 57e63aa5 >2012-11-08 09:56:02,821 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-31) [6ce786cd] FINISH, HSMClearTaskVDSCommand, log id: 57e63aa5 >2012-11-08 09:56:02,821 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-31) [6ce786cd] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@246870c7, log id: 6994fb74 >2012-11-08 09:56:02,822 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-31) [44c42b7d] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:56:02,837 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) [44c42b7d] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:56:12,896 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [337e7a4] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:56:12,922 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [337e7a4] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:56:12,924 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [337e7a4] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:56:12,936 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [337e7a4] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 32dd5a9 >2012-11-08 09:56:12,962 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [337e7a4] spmStart polling started: taskId = 16072231-0bbf-4d73-b5f1-a69eb96d8a4b >2012-11-08 09:56:13,998 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-46) [337e7a4] Failed in HSMGetTaskStatusVDS method >2012-11-08 09:56:13,998 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-46) [337e7a4] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:56:13,998 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [337e7a4] spmStart polling ended: taskId = 16072231-0bbf-4d73-b5f1-a69eb96d8a4b task status = finished >2012-11-08 09:56:13,998 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [337e7a4] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:56:14,025 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [337e7a4] spmStart polling ended, spm status: Free >2012-11-08 09:56:14,036 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-46) [337e7a4] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=16072231-0bbf-4d73-b5f1-a69eb96d8a4b), log id: 45bf3d43 >2012-11-08 09:56:14,058 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-46) [337e7a4] FINISH, HSMClearTaskVDSCommand, log id: 45bf3d43 >2012-11-08 09:56:14,058 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [337e7a4] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@e4f1d7b, log id: 32dd5a9 >2012-11-08 09:56:14,060 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-46) [6c187d93] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:56:14,084 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [6c187d93] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:56:24,143 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-91) [2ffd41b4] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:56:24,170 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-91) [2ffd41b4] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:56:24,174 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-91) [2ffd41b4] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:56:24,190 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-91) [2ffd41b4] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2b3c6a86 >2012-11-08 09:56:24,211 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-91) [2ffd41b4] spmStart polling started: taskId = 8b2c762d-1f1b-4966-9267-683f5d77f50b >2012-11-08 09:56:25,243 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-91) [2ffd41b4] Failed in HSMGetTaskStatusVDS method >2012-11-08 09:56:25,243 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-91) [2ffd41b4] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:56:25,243 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-91) [2ffd41b4] spmStart polling ended: taskId = 8b2c762d-1f1b-4966-9267-683f5d77f50b task status = finished >2012-11-08 09:56:25,243 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-91) [2ffd41b4] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:56:25,272 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-91) [2ffd41b4] spmStart polling ended, spm status: Free >2012-11-08 09:56:25,292 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-91) [2ffd41b4] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=8b2c762d-1f1b-4966-9267-683f5d77f50b), log id: 5a62871 >2012-11-08 09:56:25,311 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-91) [2ffd41b4] FINISH, HSMClearTaskVDSCommand, log id: 5a62871 >2012-11-08 09:56:25,311 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-91) [2ffd41b4] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@5fa9159c, log id: 2b3c6a86 >2012-11-08 09:56:25,314 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-91) [55c9e5d4] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:56:25,335 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-91) [55c9e5d4] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:56:28,465 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [543276f7] domain e0cce407-ebea-4322-b47e-7aceb73a7a6f:DISREP in problem. vds: rhs-gp-srv11.lab.eng.blr.redhat.com >2012-11-08 09:56:35,393 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-26) [5024f072] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:56:35,421 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-26) [5024f072] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:56:35,423 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-26) [5024f072] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:56:35,447 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-26) [5024f072] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 10d56354 >2012-11-08 09:56:35,469 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-26) [5024f072] spmStart polling started: taskId = 64f9201c-3fd1-47bd-9cf2-f420440366c6 >2012-11-08 09:56:36,501 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-26) [5024f072] Failed in HSMGetTaskStatusVDS method >2012-11-08 09:56:36,501 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-26) [5024f072] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:56:36,501 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-26) [5024f072] spmStart polling ended: taskId = 64f9201c-3fd1-47bd-9cf2-f420440366c6 task status = finished >2012-11-08 09:56:36,501 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-26) [5024f072] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:56:36,531 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-26) [5024f072] spmStart polling ended, spm status: Free >2012-11-08 09:56:36,543 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-26) [5024f072] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=64f9201c-3fd1-47bd-9cf2-f420440366c6), log id: 30c3f4c >2012-11-08 09:56:36,562 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-26) [5024f072] FINISH, HSMClearTaskVDSCommand, log id: 30c3f4c >2012-11-08 09:56:36,562 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-26) [5024f072] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@190f9824, log id: 10d56354 >2012-11-08 09:56:36,564 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-26) [4ecd7056] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:56:36,584 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-26) [4ecd7056] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:56:46,632 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [19edb1b8] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:56:46,660 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [19edb1b8] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:56:46,662 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [19edb1b8] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:56:46,679 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [19edb1b8] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 691f55a3 >2012-11-08 09:56:46,702 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [19edb1b8] spmStart polling started: taskId = f19b33ce-e334-4c22-a5e4-7847803406d3 >2012-11-08 09:56:47,733 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-72) [19edb1b8] Failed in HSMGetTaskStatusVDS method >2012-11-08 09:56:47,733 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-72) [19edb1b8] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:56:47,734 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [19edb1b8] spmStart polling ended: taskId = f19b33ce-e334-4c22-a5e4-7847803406d3 task status = finished >2012-11-08 09:56:47,734 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [19edb1b8] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:56:47,761 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [19edb1b8] spmStart polling ended, spm status: Free >2012-11-08 09:56:47,773 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-72) [19edb1b8] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=f19b33ce-e334-4c22-a5e4-7847803406d3), log id: 677017a0 >2012-11-08 09:56:47,791 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-72) [19edb1b8] FINISH, HSMClearTaskVDSCommand, log id: 677017a0 >2012-11-08 09:56:47,791 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [19edb1b8] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@701f3e6d, log id: 691f55a3 >2012-11-08 09:56:47,793 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-72) [133b9f9d] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:56:47,813 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [133b9f9d] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:56:57,872 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-89) [6380dab2] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:56:57,894 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-89) [6380dab2] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:56:57,897 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-89) [6380dab2] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:56:57,911 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-89) [6380dab2] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 295f5831 >2012-11-08 09:56:57,934 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-89) [6380dab2] spmStart polling started: taskId = ce2b8d9e-1445-4957-bba2-ba8d1db9ceae >2012-11-08 09:56:58,967 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-89) [6380dab2] Failed in HSMGetTaskStatusVDS method >2012-11-08 09:56:58,967 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-89) [6380dab2] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:56:58,967 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-89) [6380dab2] spmStart polling ended: taskId = ce2b8d9e-1445-4957-bba2-ba8d1db9ceae task status = finished >2012-11-08 09:56:58,967 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-89) [6380dab2] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:56:58,999 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-89) [6380dab2] spmStart polling ended, spm status: Free >2012-11-08 09:56:59,012 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-89) [6380dab2] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=ce2b8d9e-1445-4957-bba2-ba8d1db9ceae), log id: 31c04492 >2012-11-08 09:56:59,030 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-89) [6380dab2] FINISH, HSMClearTaskVDSCommand, log id: 31c04492 >2012-11-08 09:56:59,030 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-89) [6380dab2] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@45d5e6c2, log id: 295f5831 >2012-11-08 09:56:59,032 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-89) [5e20ee5f] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:56:59,054 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-89) [5e20ee5f] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:57:09,131 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-22) [aed4499] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:57:09,160 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-22) [aed4499] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:57:09,162 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-22) [aed4499] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:57:09,179 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-22) [aed4499] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 1ab6d8db >2012-11-08 09:57:09,200 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-22) [aed4499] spmStart polling started: taskId = 91ef3fcc-234b-47f0-ae62-6ecb11eb8853 >2012-11-08 09:57:10,231 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-22) [aed4499] Failed in HSMGetTaskStatusVDS method >2012-11-08 09:57:10,231 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-22) [aed4499] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:57:10,231 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-22) [aed4499] spmStart polling ended: taskId = 91ef3fcc-234b-47f0-ae62-6ecb11eb8853 task status = finished >2012-11-08 09:57:10,231 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-22) [aed4499] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:57:10,259 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-22) [aed4499] spmStart polling ended, spm status: Free >2012-11-08 09:57:10,277 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-22) [aed4499] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=91ef3fcc-234b-47f0-ae62-6ecb11eb8853), log id: 56fc2d27 >2012-11-08 09:57:10,294 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-22) [aed4499] FINISH, HSMClearTaskVDSCommand, log id: 56fc2d27 >2012-11-08 09:57:10,294 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-22) [aed4499] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@45f8880e, log id: 1ab6d8db >2012-11-08 09:57:10,297 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-22) [3fac7305] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:57:10,320 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-22) [3fac7305] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:57:20,386 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) [1c2d5b4] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:57:20,410 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) [1c2d5b4] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:57:20,412 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) [1c2d5b4] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:57:20,428 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-13) [1c2d5b4] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3e15ed95 >2012-11-08 09:57:20,458 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-13) [1c2d5b4] spmStart polling started: taskId = 95453500-b12a-4ff2-969d-a0b2b37c30dc >2012-11-08 09:57:21,502 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-13) [1c2d5b4] Failed in HSMGetTaskStatusVDS method >2012-11-08 09:57:21,502 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-13) [1c2d5b4] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:57:21,502 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-13) [1c2d5b4] spmStart polling ended: taskId = 95453500-b12a-4ff2-969d-a0b2b37c30dc task status = finished >2012-11-08 09:57:21,503 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-13) [1c2d5b4] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:57:21,530 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-13) [1c2d5b4] spmStart polling ended, spm status: Free >2012-11-08 09:57:21,542 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-13) [1c2d5b4] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=95453500-b12a-4ff2-969d-a0b2b37c30dc), log id: 40411137 >2012-11-08 09:57:21,565 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-13) [1c2d5b4] FINISH, HSMClearTaskVDSCommand, log id: 40411137 >2012-11-08 09:57:21,565 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-13) [1c2d5b4] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6fd152d5, log id: 3e15ed95 >2012-11-08 09:57:21,567 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-13) [1795574d] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:57:21,589 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) [1795574d] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:57:31,645 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-67) [63b6f1e9] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:57:31,675 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-67) [63b6f1e9] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:57:31,677 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-67) [63b6f1e9] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:57:31,697 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-67) [63b6f1e9] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 21c8058 >2012-11-08 09:57:31,718 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-67) [63b6f1e9] spmStart polling started: taskId = be95e9a6-ab0d-45bb-8624-1bac9d460211 >2012-11-08 09:57:32,750 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-67) [63b6f1e9] Failed in HSMGetTaskStatusVDS method >2012-11-08 09:57:32,750 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-67) [63b6f1e9] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:57:32,750 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-67) [63b6f1e9] spmStart polling ended: taskId = be95e9a6-ab0d-45bb-8624-1bac9d460211 task status = finished >2012-11-08 09:57:32,750 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-67) [63b6f1e9] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:57:32,779 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-67) [63b6f1e9] spmStart polling ended, spm status: Free >2012-11-08 09:57:32,791 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-67) [63b6f1e9] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=be95e9a6-ab0d-45bb-8624-1bac9d460211), log id: 6118cd2e >2012-11-08 09:57:32,810 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-67) [63b6f1e9] FINISH, HSMClearTaskVDSCommand, log id: 6118cd2e >2012-11-08 09:57:32,810 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-67) [63b6f1e9] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2346b8cf, log id: 21c8058 >2012-11-08 09:57:32,811 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-67) [780f3020] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:57:32,825 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-67) [780f3020] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:57:42,875 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-19) [6c49c0f] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:57:42,905 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-19) [6c49c0f] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:57:42,908 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-19) [6c49c0f] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:57:42,925 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-19) [6c49c0f] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 12f6c759 >2012-11-08 09:57:42,950 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-19) [6c49c0f] spmStart polling started: taskId = a59e529b-6234-4016-9b00-9518ef1dd80c >2012-11-08 09:57:43,986 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-19) [6c49c0f] Failed in HSMGetTaskStatusVDS method >2012-11-08 09:57:43,986 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-19) [6c49c0f] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:57:43,987 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-19) [6c49c0f] spmStart polling ended: taskId = a59e529b-6234-4016-9b00-9518ef1dd80c task status = finished >2012-11-08 09:57:43,987 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-19) [6c49c0f] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:57:44,014 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-19) [6c49c0f] spmStart polling ended, spm status: Free >2012-11-08 09:57:44,032 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-19) [6c49c0f] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=a59e529b-6234-4016-9b00-9518ef1dd80c), log id: 2f0ee56f >2012-11-08 09:57:44,060 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-19) [6c49c0f] FINISH, HSMClearTaskVDSCommand, log id: 2f0ee56f >2012-11-08 09:57:44,060 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-19) [6c49c0f] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7af8f28b, log id: 12f6c759 >2012-11-08 09:57:44,062 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-19) [6dfbc2bf] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:57:44,084 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-19) [6dfbc2bf] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:57:54,140 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [6c187d93] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:57:54,169 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [6c187d93] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:57:54,172 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [6c187d93] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:57:54,187 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [6c187d93] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: a6d8bdc >2012-11-08 09:57:54,212 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [6c187d93] spmStart polling started: taskId = fa67d5be-96b6-4c02-9af4-c77db26c41df >2012-11-08 09:57:55,252 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-46) [6c187d93] Failed in HSMGetTaskStatusVDS method >2012-11-08 09:57:55,252 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-46) [6c187d93] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:57:55,252 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [6c187d93] spmStart polling ended: taskId = fa67d5be-96b6-4c02-9af4-c77db26c41df task status = finished >2012-11-08 09:57:55,252 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [6c187d93] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:57:55,279 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [6c187d93] spmStart polling ended, spm status: Free >2012-11-08 09:57:55,291 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-46) [6c187d93] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=fa67d5be-96b6-4c02-9af4-c77db26c41df), log id: 7ae8aa6b >2012-11-08 09:57:55,308 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-46) [6c187d93] FINISH, HSMClearTaskVDSCommand, log id: 7ae8aa6b >2012-11-08 09:57:55,309 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [6c187d93] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@410c95d7, log id: a6d8bdc >2012-11-08 09:57:55,311 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-46) [2141779e] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:57:55,336 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [2141779e] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:58:05,386 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-91) [55c9e5d4] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:58:05,411 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-91) [55c9e5d4] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:58:05,414 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-91) [55c9e5d4] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:58:05,428 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-91) [55c9e5d4] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3e7cadd2 >2012-11-08 09:58:05,450 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-91) [55c9e5d4] spmStart polling started: taskId = dde63da4-456d-476a-ae30-1a6598d28830 >2012-11-08 09:58:06,480 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-91) [55c9e5d4] Failed in HSMGetTaskStatusVDS method >2012-11-08 09:58:06,480 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-91) [55c9e5d4] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:58:06,480 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-91) [55c9e5d4] spmStart polling ended: taskId = dde63da4-456d-476a-ae30-1a6598d28830 task status = finished >2012-11-08 09:58:06,480 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-91) [55c9e5d4] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:58:06,509 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-91) [55c9e5d4] spmStart polling ended, spm status: Free >2012-11-08 09:58:06,520 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-91) [55c9e5d4] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=dde63da4-456d-476a-ae30-1a6598d28830), log id: 6e4fde96 >2012-11-08 09:58:06,538 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-91) [55c9e5d4] FINISH, HSMClearTaskVDSCommand, log id: 6e4fde96 >2012-11-08 09:58:06,538 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-91) [55c9e5d4] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3d9e826e, log id: 3e7cadd2 >2012-11-08 09:58:06,541 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-91) [7185235c] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:58:06,560 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-91) [7185235c] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:58:16,626 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) [44c42b7d] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:58:16,671 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) [44c42b7d] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:58:16,673 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) [44c42b7d] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:58:16,692 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-31) [44c42b7d] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6bdefeba >2012-11-08 09:58:16,723 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-31) [44c42b7d] spmStart polling started: taskId = 1570d3af-03c7-4fed-b392-097b3cbf07ef >2012-11-08 09:58:17,775 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-31) [44c42b7d] Failed in HSMGetTaskStatusVDS method >2012-11-08 09:58:17,776 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-31) [44c42b7d] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:58:17,776 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-31) [44c42b7d] spmStart polling ended: taskId = 1570d3af-03c7-4fed-b392-097b3cbf07ef task status = finished >2012-11-08 09:58:17,776 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-31) [44c42b7d] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:58:17,813 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-31) [44c42b7d] spmStart polling ended, spm status: Free >2012-11-08 09:58:17,832 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-31) [44c42b7d] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=1570d3af-03c7-4fed-b392-097b3cbf07ef), log id: 7457a6c8 >2012-11-08 09:58:17,849 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-31) [44c42b7d] FINISH, HSMClearTaskVDSCommand, log id: 7457a6c8 >2012-11-08 09:58:17,849 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-31) [44c42b7d] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2686d024, log id: 6bdefeba >2012-11-08 09:58:17,851 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-31) [5ccaff3e] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:58:17,875 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-31) [5ccaff3e] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:58:27,935 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) [1795574d] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:58:27,963 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) [1795574d] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:58:27,966 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) [1795574d] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:58:27,984 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-13) [1795574d] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7ff42f33 >2012-11-08 09:58:28,005 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-13) [1795574d] spmStart polling started: taskId = d0246b76-7f28-4e8f-8b5e-f47cc174f634 >2012-11-08 09:58:29,037 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-13) [1795574d] Failed in HSMGetTaskStatusVDS method >2012-11-08 09:58:29,037 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-13) [1795574d] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:58:29,037 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-13) [1795574d] spmStart polling ended: taskId = d0246b76-7f28-4e8f-8b5e-f47cc174f634 task status = finished >2012-11-08 09:58:29,037 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-13) [1795574d] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:58:29,077 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-13) [1795574d] spmStart polling ended, spm status: Free >2012-11-08 09:58:29,094 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-13) [1795574d] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=d0246b76-7f28-4e8f-8b5e-f47cc174f634), log id: 3c767a29 >2012-11-08 09:58:29,111 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-13) [1795574d] FINISH, HSMClearTaskVDSCommand, log id: 3c767a29 >2012-11-08 09:58:29,111 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-13) [1795574d] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@64e38c2, log id: 7ff42f33 >2012-11-08 09:58:29,114 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-13) [79704bc9] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:58:29,136 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) [79704bc9] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:58:39,250 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [72878333] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:58:39,288 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [72878333] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:58:39,290 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [72878333] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:58:39,306 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-84) [72878333] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 108a36aa >2012-11-08 09:58:39,328 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-84) [72878333] spmStart polling started: taskId = 8815d21c-4b91-41f8-b4ad-a5db4dc46a30 >2012-11-08 09:58:40,357 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-84) [72878333] Failed in HSMGetTaskStatusVDS method >2012-11-08 09:58:40,357 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-84) [72878333] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:58:40,357 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-84) [72878333] spmStart polling ended: taskId = 8815d21c-4b91-41f8-b4ad-a5db4dc46a30 task status = finished >2012-11-08 09:58:40,357 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-84) [72878333] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:58:40,384 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-84) [72878333] spmStart polling ended, spm status: Free >2012-11-08 09:58:40,397 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-84) [72878333] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=8815d21c-4b91-41f8-b4ad-a5db4dc46a30), log id: 4e8f4f8 >2012-11-08 09:58:40,414 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-84) [72878333] FINISH, HSMClearTaskVDSCommand, log id: 4e8f4f8 >2012-11-08 09:58:40,414 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-84) [72878333] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1f4c5192, log id: 108a36aa >2012-11-08 09:58:40,416 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-84) [4b7f584b] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:58:40,433 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [4b7f584b] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:58:50,492 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-22) [3fac7305] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:58:50,522 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-22) [3fac7305] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:58:50,525 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-22) [3fac7305] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:58:50,542 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-22) [3fac7305] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 48693b55 >2012-11-08 09:58:50,567 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-22) [3fac7305] spmStart polling started: taskId = 2601faf5-4f82-443c-9613-07bea9e13d1f >2012-11-08 09:58:51,601 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-22) [3fac7305] Failed in HSMGetTaskStatusVDS method >2012-11-08 09:58:51,601 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-22) [3fac7305] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:58:51,601 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-22) [3fac7305] spmStart polling ended: taskId = 2601faf5-4f82-443c-9613-07bea9e13d1f task status = finished >2012-11-08 09:58:51,601 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-22) [3fac7305] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:58:51,635 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-22) [3fac7305] spmStart polling ended, spm status: Free >2012-11-08 09:58:51,653 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-22) [3fac7305] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=2601faf5-4f82-443c-9613-07bea9e13d1f), log id: 31db72d4 >2012-11-08 09:58:51,671 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-22) [3fac7305] FINISH, HSMClearTaskVDSCommand, log id: 31db72d4 >2012-11-08 09:58:51,671 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-22) [3fac7305] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3f1cea3f, log id: 48693b55 >2012-11-08 09:58:51,673 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-22) [356ee78] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:58:51,701 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-22) [356ee78] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:59:01,757 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-93) [34850295] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:59:01,784 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-93) [34850295] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:59:01,787 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-93) [34850295] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:59:01,810 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-93) [34850295] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 609d2afe >2012-11-08 09:59:01,835 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-93) [34850295] spmStart polling started: taskId = 2fded3fe-4c54-49b4-8a4d-fa61aabfb18b >2012-11-08 09:59:02,877 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-93) [34850295] Failed in HSMGetTaskStatusVDS method >2012-11-08 09:59:02,878 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-93) [34850295] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:59:02,878 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-93) [34850295] spmStart polling ended: taskId = 2fded3fe-4c54-49b4-8a4d-fa61aabfb18b task status = finished >2012-11-08 09:59:02,878 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-93) [34850295] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:59:02,906 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-93) [34850295] spmStart polling ended, spm status: Free >2012-11-08 09:59:02,917 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-93) [34850295] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=2fded3fe-4c54-49b4-8a4d-fa61aabfb18b), log id: 5ea2444b >2012-11-08 09:59:02,939 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-93) [34850295] FINISH, HSMClearTaskVDSCommand, log id: 5ea2444b >2012-11-08 09:59:02,939 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-93) [34850295] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6f31c876, log id: 609d2afe >2012-11-08 09:59:02,941 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-93) [2d5ff402] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:59:02,958 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-93) [2d5ff402] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:59:13,017 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-67) [780f3020] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:59:13,044 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-67) [780f3020] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:59:13,046 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-67) [780f3020] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:59:13,062 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-67) [780f3020] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 718b9070 >2012-11-08 09:59:13,084 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-67) [780f3020] spmStart polling started: taskId = f1c1bfd5-afc6-4431-9165-63c869d79b18 >2012-11-08 09:59:14,115 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-67) [780f3020] Failed in HSMGetTaskStatusVDS method >2012-11-08 09:59:14,115 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-67) [780f3020] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:59:14,115 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-67) [780f3020] spmStart polling ended: taskId = f1c1bfd5-afc6-4431-9165-63c869d79b18 task status = finished >2012-11-08 09:59:14,115 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-67) [780f3020] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:59:14,144 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-67) [780f3020] spmStart polling ended, spm status: Free >2012-11-08 09:59:14,157 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-67) [780f3020] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=f1c1bfd5-afc6-4431-9165-63c869d79b18), log id: 40b80adc >2012-11-08 09:59:14,180 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-67) [780f3020] FINISH, HSMClearTaskVDSCommand, log id: 40b80adc >2012-11-08 09:59:14,180 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-67) [780f3020] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7777eb25, log id: 718b9070 >2012-11-08 09:59:14,183 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-67) [61ef0769] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:59:14,205 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-67) [61ef0769] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:59:24,266 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:59:24,294 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:59:24,297 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:59:24,313 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6e816a >2012-11-08 09:59:24,342 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) spmStart polling started: taskId = db95d6ad-26cc-4fd3-a730-f9eeda8514b6 >2012-11-08 09:59:25,382 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-48) Failed in HSMGetTaskStatusVDS method >2012-11-08 09:59:25,382 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-48) Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:59:25,382 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) spmStart polling ended: taskId = db95d6ad-26cc-4fd3-a730-f9eeda8514b6 task status = finished >2012-11-08 09:59:25,382 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:59:25,412 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) spmStart polling ended, spm status: Free >2012-11-08 09:59:25,425 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-48) START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=db95d6ad-26cc-4fd3-a730-f9eeda8514b6), log id: 52b043b4 >2012-11-08 09:59:25,448 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-48) FINISH, HSMClearTaskVDSCommand, log id: 52b043b4 >2012-11-08 09:59:25,448 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@322d08d5, log id: 6e816a >2012-11-08 09:59:25,453 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-48) [2a9ef1af] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:59:25,467 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [2a9ef1af] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:59:35,521 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) [79704bc9] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:59:35,551 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) [79704bc9] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:59:35,553 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) [79704bc9] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:59:35,571 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-13) [79704bc9] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7b92d46 >2012-11-08 09:59:35,601 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-13) [79704bc9] spmStart polling started: taskId = d83a5701-29cd-490d-be4a-d485740bd86c >2012-11-08 09:59:36,646 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-13) [79704bc9] Failed in HSMGetTaskStatusVDS method >2012-11-08 09:59:36,646 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-13) [79704bc9] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:59:36,646 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-13) [79704bc9] spmStart polling ended: taskId = d83a5701-29cd-490d-be4a-d485740bd86c task status = finished >2012-11-08 09:59:36,646 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-13) [79704bc9] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:59:36,675 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-13) [79704bc9] spmStart polling ended, spm status: Free >2012-11-08 09:59:36,692 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-13) [79704bc9] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=d83a5701-29cd-490d-be4a-d485740bd86c), log id: 7a2aa27a >2012-11-08 09:59:36,716 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-13) [79704bc9] FINISH, HSMClearTaskVDSCommand, log id: 7a2aa27a >2012-11-08 09:59:36,716 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-13) [79704bc9] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@511363f, log id: 7b92d46 >2012-11-08 09:59:36,718 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-13) [2d0064b2] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:59:36,735 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) [2d0064b2] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:59:46,797 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-6) [2ed69c19] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:59:46,824 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-6) [2ed69c19] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:59:46,826 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-6) [2ed69c19] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:59:46,842 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-6) [2ed69c19] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 45698db6 >2012-11-08 09:59:46,866 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-6) [2ed69c19] spmStart polling started: taskId = 989c808d-08c3-4a2c-acea-65611be51e5d >2012-11-08 09:59:47,900 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-6) [2ed69c19] Failed in HSMGetTaskStatusVDS method >2012-11-08 09:59:47,922 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-6) [2ed69c19] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:59:47,922 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-6) [2ed69c19] spmStart polling ended: taskId = 989c808d-08c3-4a2c-acea-65611be51e5d task status = finished >2012-11-08 09:59:47,923 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-6) [2ed69c19] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:59:47,952 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-6) [2ed69c19] spmStart polling ended, spm status: Free >2012-11-08 09:59:47,965 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-6) [2ed69c19] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=989c808d-08c3-4a2c-acea-65611be51e5d), log id: 6976efdc >2012-11-08 09:59:47,987 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-6) [2ed69c19] FINISH, HSMClearTaskVDSCommand, log id: 6976efdc >2012-11-08 09:59:47,987 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-6) [2ed69c19] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7f7d8a9b, log id: 45698db6 >2012-11-08 09:59:47,989 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-6) [b170781] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:59:48,007 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-6) [b170781] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 09:59:58,060 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-22) [356ee78] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 09:59:58,089 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-22) [356ee78] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 09:59:58,092 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-22) [356ee78] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 09:59:58,109 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-22) [356ee78] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3514e29b >2012-11-08 09:59:58,131 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-22) [356ee78] spmStart polling started: taskId = 8e6e5362-5e88-468f-913e-55807183cadc >2012-11-08 09:59:59,166 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-22) [356ee78] Failed in HSMGetTaskStatusVDS method >2012-11-08 09:59:59,166 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-22) [356ee78] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:59:59,166 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-22) [356ee78] spmStart polling ended: taskId = 8e6e5362-5e88-468f-913e-55807183cadc task status = finished >2012-11-08 09:59:59,166 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-22) [356ee78] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 09:59:59,192 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-22) [356ee78] spmStart polling ended, spm status: Free >2012-11-08 09:59:59,209 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-22) [356ee78] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=8e6e5362-5e88-468f-913e-55807183cadc), log id: 45aeddcf >2012-11-08 09:59:59,227 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-22) [356ee78] FINISH, HSMClearTaskVDSCommand, log id: 45aeddcf >2012-11-08 09:59:59,227 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-22) [356ee78] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@5b6464c6, log id: 3514e29b >2012-11-08 09:59:59,229 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-22) [1c14320a] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 09:59:59,254 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-22) [1c14320a] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:00:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-29) [251cffc8] Autorecovering hosts is disabled, skipping >2012-11-08 10:00:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-29) [251cffc8] Autorecovering storage domains is disabled, skipping >2012-11-08 10:00:09,315 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [5e0e7a4] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:00:09,341 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [5e0e7a4] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:00:09,343 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [5e0e7a4] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:00:09,360 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-74) [5e0e7a4] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5dd02a79 >2012-11-08 10:00:09,389 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-74) [5e0e7a4] spmStart polling started: taskId = 4cfef7a4-7b95-421d-8c63-f6ad5aea7aa2 >2012-11-08 10:00:10,426 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-74) [5e0e7a4] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:00:10,426 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-74) [5e0e7a4] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:00:10,426 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-74) [5e0e7a4] spmStart polling ended: taskId = 4cfef7a4-7b95-421d-8c63-f6ad5aea7aa2 task status = finished >2012-11-08 10:00:10,426 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-74) [5e0e7a4] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:00:10,457 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-74) [5e0e7a4] spmStart polling ended, spm status: Free >2012-11-08 10:00:10,472 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-74) [5e0e7a4] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=4cfef7a4-7b95-421d-8c63-f6ad5aea7aa2), log id: 493148c7 >2012-11-08 10:00:10,495 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-74) [5e0e7a4] FINISH, HSMClearTaskVDSCommand, log id: 493148c7 >2012-11-08 10:00:10,495 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-74) [5e0e7a4] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@42a1265b, log id: 5dd02a79 >2012-11-08 10:00:10,497 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-74) [61dd4289] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:00:10,513 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-74) [61dd4289] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:00:20,572 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-12) hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:00:20,599 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-12) SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:00:20,601 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-12) starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:00:20,613 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-12) START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 79fcf642 >2012-11-08 10:00:20,637 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-12) spmStart polling started: taskId = 2932e7b5-898a-4bbd-9594-3e6fbc79d860 >2012-11-08 10:00:21,678 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-12) Failed in HSMGetTaskStatusVDS method >2012-11-08 10:00:21,678 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-12) Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:00:21,678 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-12) spmStart polling ended: taskId = 2932e7b5-898a-4bbd-9594-3e6fbc79d860 task status = finished >2012-11-08 10:00:21,678 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-12) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:00:21,707 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-12) spmStart polling ended, spm status: Free >2012-11-08 10:00:21,722 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-12) START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=2932e7b5-898a-4bbd-9594-3e6fbc79d860), log id: 3b6c015 >2012-11-08 10:00:21,739 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-12) FINISH, HSMClearTaskVDSCommand, log id: 3b6c015 >2012-11-08 10:00:21,739 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-12) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@52fca49a, log id: 79fcf642 >2012-11-08 10:00:21,741 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-12) [489aa506] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:00:21,762 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-12) [489aa506] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:00:31,820 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-47) [5f698c14] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:00:31,848 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-47) [5f698c14] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:00:31,851 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-47) [5f698c14] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:00:31,868 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-47) [5f698c14] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3e967756 >2012-11-08 10:00:31,891 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-47) [5f698c14] spmStart polling started: taskId = 39d44d25-706e-4ec0-9e7f-4e5993e9c9aa >2012-11-08 10:00:32,922 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-47) [5f698c14] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:00:32,922 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-47) [5f698c14] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:00:32,923 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-47) [5f698c14] spmStart polling ended: taskId = 39d44d25-706e-4ec0-9e7f-4e5993e9c9aa task status = finished >2012-11-08 10:00:32,923 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-47) [5f698c14] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:00:32,946 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-47) [5f698c14] spmStart polling ended, spm status: Free >2012-11-08 10:00:32,965 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-47) [5f698c14] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=39d44d25-706e-4ec0-9e7f-4e5993e9c9aa), log id: 5d19718d >2012-11-08 10:00:32,987 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-47) [5f698c14] FINISH, HSMClearTaskVDSCommand, log id: 5d19718d >2012-11-08 10:00:32,987 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-47) [5f698c14] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@316e0fbb, log id: 3e967756 >2012-11-08 10:00:32,989 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-47) [5ce37369] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:00:33,007 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-47) [5ce37369] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:00:43,057 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [445b6a4e] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:00:43,088 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [445b6a4e] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:00:43,090 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [445b6a4e] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:00:43,107 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-73) [445b6a4e] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6da18eed >2012-11-08 10:00:43,129 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-73) [445b6a4e] spmStart polling started: taskId = 5a08632d-7fb9-49f5-9a28-05279ad15386 >2012-11-08 10:00:44,161 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-73) [445b6a4e] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:00:44,161 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-73) [445b6a4e] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:00:44,161 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-73) [445b6a4e] spmStart polling ended: taskId = 5a08632d-7fb9-49f5-9a28-05279ad15386 task status = finished >2012-11-08 10:00:44,161 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-73) [445b6a4e] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:00:44,190 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-73) [445b6a4e] spmStart polling ended, spm status: Free >2012-11-08 10:00:44,208 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-73) [445b6a4e] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=5a08632d-7fb9-49f5-9a28-05279ad15386), log id: 7d987756 >2012-11-08 10:00:44,233 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-73) [445b6a4e] FINISH, HSMClearTaskVDSCommand, log id: 7d987756 >2012-11-08 10:00:44,233 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-73) [445b6a4e] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@72a99a15, log id: 6da18eed >2012-11-08 10:00:44,235 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-73) [5cc31382] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:00:44,261 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [5cc31382] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:00:54,316 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [543276f7] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:00:54,346 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [543276f7] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:00:54,349 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [543276f7] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:00:54,366 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [543276f7] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 53c886ee >2012-11-08 10:00:54,389 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [543276f7] spmStart polling started: taskId = a2bef760-c0c1-4f7c-a089-76900bdc6ae0 >2012-11-08 10:00:55,424 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-16) [543276f7] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:00:55,424 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-16) [543276f7] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:00:55,424 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [543276f7] spmStart polling ended: taskId = a2bef760-c0c1-4f7c-a089-76900bdc6ae0 task status = finished >2012-11-08 10:00:55,424 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [543276f7] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:00:55,456 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [543276f7] spmStart polling ended, spm status: Free >2012-11-08 10:00:55,475 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-16) [543276f7] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=a2bef760-c0c1-4f7c-a089-76900bdc6ae0), log id: 3cab12bb >2012-11-08 10:00:55,493 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-16) [543276f7] FINISH, HSMClearTaskVDSCommand, log id: 3cab12bb >2012-11-08 10:00:55,493 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [543276f7] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@35ad21b, log id: 53c886ee >2012-11-08 10:00:55,495 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-16) [3f97ec55] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:00:55,519 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [3f97ec55] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:01:05,581 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-22) [1c14320a] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:01:05,607 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-22) [1c14320a] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:01:05,610 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-22) [1c14320a] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:01:05,627 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-22) [1c14320a] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 22dbe4e2 >2012-11-08 10:01:05,656 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-22) [1c14320a] spmStart polling started: taskId = 2376e8b2-2120-4f76-b66d-601e333d93a2 >2012-11-08 10:01:06,694 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-22) [1c14320a] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:01:06,694 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-22) [1c14320a] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:01:06,694 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-22) [1c14320a] spmStart polling ended: taskId = 2376e8b2-2120-4f76-b66d-601e333d93a2 task status = finished >2012-11-08 10:01:06,694 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-22) [1c14320a] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:01:06,725 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-22) [1c14320a] spmStart polling ended, spm status: Free >2012-11-08 10:01:06,739 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-22) [1c14320a] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=2376e8b2-2120-4f76-b66d-601e333d93a2), log id: 6854fb9d >2012-11-08 10:01:06,763 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-22) [1c14320a] FINISH, HSMClearTaskVDSCommand, log id: 6854fb9d >2012-11-08 10:01:06,763 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-22) [1c14320a] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@235335ee, log id: 22dbe4e2 >2012-11-08 10:01:06,765 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-22) [2bf9ae60] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:01:06,791 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-22) [2bf9ae60] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:01:16,852 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-80) [3dbe1e04] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:01:16,879 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-80) [3dbe1e04] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:01:16,881 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-80) [3dbe1e04] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:01:16,897 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-80) [3dbe1e04] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: e53df11 >2012-11-08 10:01:16,920 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-80) [3dbe1e04] spmStart polling started: taskId = 59b238c1-2b06-4f63-bfe2-5b3e4acbe8b0 >2012-11-08 10:01:17,950 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-80) [3dbe1e04] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:01:17,951 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-80) [3dbe1e04] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:01:17,951 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-80) [3dbe1e04] spmStart polling ended: taskId = 59b238c1-2b06-4f63-bfe2-5b3e4acbe8b0 task status = finished >2012-11-08 10:01:17,951 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-80) [3dbe1e04] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:01:17,978 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-80) [3dbe1e04] spmStart polling ended, spm status: Free >2012-11-08 10:01:17,999 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-80) [3dbe1e04] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=59b238c1-2b06-4f63-bfe2-5b3e4acbe8b0), log id: bf70a25 >2012-11-08 10:01:18,016 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-80) [3dbe1e04] FINISH, HSMClearTaskVDSCommand, log id: bf70a25 >2012-11-08 10:01:18,017 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-80) [3dbe1e04] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7bb0946, log id: e53df11 >2012-11-08 10:01:18,018 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-80) [85dd6a1] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:01:18,042 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-80) [85dd6a1] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:01:28,103 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-11) [4c0adb54] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:01:28,129 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-11) [4c0adb54] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:01:28,131 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-11) [4c0adb54] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:01:28,147 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-11) [4c0adb54] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5a24e7cd >2012-11-08 10:01:28,171 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-11) [4c0adb54] spmStart polling started: taskId = 20cebf7a-1c21-4ee0-8d10-b0d2bcc1d65a >2012-11-08 10:01:28,466 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [3f97ec55] starting ProcessDomainRecovery for domain e0cce407-ebea-4322-b47e-7aceb73a7a6f:DISREP >2012-11-08 10:01:28,505 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [3f97ec55] domain e0cce407-ebea-4322-b47e-7aceb73a7a6f:DISREP was reported by all hosts in status UP as problematic. Moving the Domain to NonOperational. >2012-11-08 10:01:28,522 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (QuartzScheduler_Worker-16) [60ad9f29] Lock Acquired to object EngineLock [exclusiveLocks= key: e0cce407-ebea-4322-b47e-7aceb73a7a6f value: STORAGE >, sharedLocks= key: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 value: POOL >] >2012-11-08 10:01:28,522 WARN [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (QuartzScheduler_Worker-16) [60ad9f29] CanDoAction of action DeactivateStorageDomain failed. Reasons:VAR__TYPE__STORAGE__DOMAIN,VAR__ACTION__DEACTIVATE,ACTION_TYPE_FAILED_STORAGE_DOMAIN_STATUS_ILLEGAL >2012-11-08 10:01:28,522 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (QuartzScheduler_Worker-16) [60ad9f29] Lock freed to object EngineLock [exclusiveLocks= key: e0cce407-ebea-4322-b47e-7aceb73a7a6f value: STORAGE >, sharedLocks= key: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 value: POOL >] >2012-11-08 10:01:29,213 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-11) [4c0adb54] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:01:29,213 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-11) [4c0adb54] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:01:29,213 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-11) [4c0adb54] spmStart polling ended: taskId = 20cebf7a-1c21-4ee0-8d10-b0d2bcc1d65a task status = finished >2012-11-08 10:01:29,214 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-11) [4c0adb54] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:01:29,240 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-11) [4c0adb54] spmStart polling ended, spm status: Free >2012-11-08 10:01:29,251 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-11) [4c0adb54] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=20cebf7a-1c21-4ee0-8d10-b0d2bcc1d65a), log id: 5fda62d8 >2012-11-08 10:01:29,274 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-11) [4c0adb54] FINISH, HSMClearTaskVDSCommand, log id: 5fda62d8 >2012-11-08 10:01:29,274 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-11) [4c0adb54] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7a8f8411, log id: 5a24e7cd >2012-11-08 10:01:29,275 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-11) [6427b6f3] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:01:29,292 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-11) [6427b6f3] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:01:37,974 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-40) [17449a1f] domain e0cce407-ebea-4322-b47e-7aceb73a7a6f:DISREP in problem. vds: rhs-gp-srv11.lab.eng.blr.redhat.com >2012-11-08 10:01:39,358 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-29) [251cffc8] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:01:39,380 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-29) [251cffc8] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:01:39,382 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-29) [251cffc8] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:01:39,397 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-29) [251cffc8] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 12f52eab >2012-11-08 10:01:39,426 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-29) [251cffc8] spmStart polling started: taskId = 249037f5-ac24-4478-8a89-0723178fe0d6 >2012-11-08 10:01:40,462 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-29) [251cffc8] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:01:40,463 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-29) [251cffc8] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:01:40,463 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-29) [251cffc8] spmStart polling ended: taskId = 249037f5-ac24-4478-8a89-0723178fe0d6 task status = finished >2012-11-08 10:01:40,463 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-29) [251cffc8] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:01:40,494 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-29) [251cffc8] spmStart polling ended, spm status: Free >2012-11-08 10:01:40,508 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-29) [251cffc8] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=249037f5-ac24-4478-8a89-0723178fe0d6), log id: 3fb96ab0 >2012-11-08 10:01:40,531 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-29) [251cffc8] FINISH, HSMClearTaskVDSCommand, log id: 3fb96ab0 >2012-11-08 10:01:40,531 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-29) [251cffc8] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@260b697f, log id: 12f52eab >2012-11-08 10:01:40,533 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-29) [373b245c] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:01:40,549 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-29) [373b245c] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:01:50,608 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [5cc31382] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:01:50,631 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [5cc31382] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:01:50,633 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [5cc31382] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:01:50,652 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-73) [5cc31382] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 216621c0 >2012-11-08 10:01:50,678 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-73) [5cc31382] spmStart polling started: taskId = f0ed05a0-4080-4e4f-9690-ceb060d65dc4 >2012-11-08 10:01:51,713 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-73) [5cc31382] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:01:51,713 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-73) [5cc31382] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:01:51,713 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-73) [5cc31382] spmStart polling ended: taskId = f0ed05a0-4080-4e4f-9690-ceb060d65dc4 task status = finished >2012-11-08 10:01:51,713 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-73) [5cc31382] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:01:51,744 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-73) [5cc31382] spmStart polling ended, spm status: Free >2012-11-08 10:01:51,757 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-73) [5cc31382] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=f0ed05a0-4080-4e4f-9690-ceb060d65dc4), log id: 7435f0a5 >2012-11-08 10:01:51,781 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-73) [5cc31382] FINISH, HSMClearTaskVDSCommand, log id: 7435f0a5 >2012-11-08 10:01:51,781 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-73) [5cc31382] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@14a1c3e, log id: 216621c0 >2012-11-08 10:01:51,783 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-73) [28842b1b] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:01:51,800 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [28842b1b] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:02:01,862 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) [1ae03e41] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:02:01,892 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) [1ae03e41] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:02:01,894 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) [1ae03e41] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:02:01,912 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-10) [1ae03e41] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: b4c471e >2012-11-08 10:02:01,934 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-10) [1ae03e41] spmStart polling started: taskId = 11902c55-8d9e-409a-95a1-b57eda9133d5 >2012-11-08 10:02:02,967 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-10) [1ae03e41] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:02:02,967 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-10) [1ae03e41] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:02:02,968 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-10) [1ae03e41] spmStart polling ended: taskId = 11902c55-8d9e-409a-95a1-b57eda9133d5 task status = finished >2012-11-08 10:02:02,968 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-10) [1ae03e41] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:02:02,997 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-10) [1ae03e41] spmStart polling ended, spm status: Free >2012-11-08 10:02:03,014 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-10) [1ae03e41] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=11902c55-8d9e-409a-95a1-b57eda9133d5), log id: 75f75b40 >2012-11-08 10:02:03,031 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-10) [1ae03e41] FINISH, HSMClearTaskVDSCommand, log id: 75f75b40 >2012-11-08 10:02:03,031 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-10) [1ae03e41] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@42880341, log id: b4c471e >2012-11-08 10:02:03,033 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-10) [6f8dc286] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:02:03,052 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-10) [6f8dc286] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:02:13,109 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-54) hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:02:13,136 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-54) SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:02:13,138 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-54) starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:02:13,154 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 437c1fe7 >2012-11-08 10:02:13,184 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) spmStart polling started: taskId = d9ddc344-ef6a-44ec-8773-bf995b165295 >2012-11-08 10:02:14,221 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-54) Failed in HSMGetTaskStatusVDS method >2012-11-08 10:02:14,221 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-54) Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:02:14,221 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) spmStart polling ended: taskId = d9ddc344-ef6a-44ec-8773-bf995b165295 task status = finished >2012-11-08 10:02:14,221 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:02:14,253 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) spmStart polling ended, spm status: Free >2012-11-08 10:02:14,267 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-54) START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=d9ddc344-ef6a-44ec-8773-bf995b165295), log id: 6d4ab0b4 >2012-11-08 10:02:14,290 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-54) FINISH, HSMClearTaskVDSCommand, log id: 6d4ab0b4 >2012-11-08 10:02:14,290 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4297317a, log id: 437c1fe7 >2012-11-08 10:02:14,292 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-54) [543162bd] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:02:14,319 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-54) [543162bd] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:02:20,445 INFO [org.ovirt.engine.core.bll.SetVmTicketCommand] (ajp-/127.0.0.1:8702-12) [9618c0] Running command: SetVmTicketCommand internal: false. Entities affected : ID: 28f91cc7-0841-425d-a0fd-8ea1fd15c636 Type: VM >2012-11-08 10:02:20,469 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp-/127.0.0.1:8702-12) [9618c0] START, SetVmTicketVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=28f91cc7-0841-425d-a0fd-8ea1fd15c636, ticket=XupnorOI0ncn, validTime=120,m userName=admin@internal, userId=fdfc627c-d875-11e0-90f0-83df133b58cc), log id: 6e86f9d1 >2012-11-08 10:02:20,503 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp-/127.0.0.1:8702-12) [9618c0] FINISH, SetVmTicketVDSCommand, log id: 6e86f9d1 >2012-11-08 10:02:24,378 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-80) [85dd6a1] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:02:24,405 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-80) [85dd6a1] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:02:24,407 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-80) [85dd6a1] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:02:24,423 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-80) [85dd6a1] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 52c0f948 >2012-11-08 10:02:24,445 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-80) [85dd6a1] spmStart polling started: taskId = 26c7157f-d660-42c7-beaa-acec75a938d6 >2012-11-08 10:02:25,476 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-80) [85dd6a1] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:02:25,476 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-80) [85dd6a1] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:02:25,476 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-80) [85dd6a1] spmStart polling ended: taskId = 26c7157f-d660-42c7-beaa-acec75a938d6 task status = finished >2012-11-08 10:02:25,476 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-80) [85dd6a1] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:02:25,505 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-80) [85dd6a1] spmStart polling ended, spm status: Free >2012-11-08 10:02:25,518 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-80) [85dd6a1] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=26c7157f-d660-42c7-beaa-acec75a938d6), log id: 6bcd0e98 >2012-11-08 10:02:25,537 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-80) [85dd6a1] FINISH, HSMClearTaskVDSCommand, log id: 6bcd0e98 >2012-11-08 10:02:25,537 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-80) [85dd6a1] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@63795472, log id: 52c0f948 >2012-11-08 10:02:25,539 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-80) [32973b63] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:02:25,571 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-80) [32973b63] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:02:35,628 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-11) [6427b6f3] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:02:35,654 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-11) [6427b6f3] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:02:35,656 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-11) [6427b6f3] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:02:35,674 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-11) [6427b6f3] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 439dacc0 >2012-11-08 10:02:35,698 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-11) [6427b6f3] spmStart polling started: taskId = 7dc797b3-fb7e-4efd-95b6-e819b1d0797e >2012-11-08 10:02:36,731 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-11) [6427b6f3] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:02:36,731 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-11) [6427b6f3] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:02:36,731 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-11) [6427b6f3] spmStart polling ended: taskId = 7dc797b3-fb7e-4efd-95b6-e819b1d0797e task status = finished >2012-11-08 10:02:36,731 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-11) [6427b6f3] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:02:36,762 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-11) [6427b6f3] spmStart polling ended, spm status: Free >2012-11-08 10:02:36,775 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-11) [6427b6f3] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=7dc797b3-fb7e-4efd-95b6-e819b1d0797e), log id: 75ec88b3 >2012-11-08 10:02:36,799 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-11) [6427b6f3] FINISH, HSMClearTaskVDSCommand, log id: 75ec88b3 >2012-11-08 10:02:36,799 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-11) [6427b6f3] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7c02dc40, log id: 439dacc0 >2012-11-08 10:02:36,801 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-11) [1fca2d5b] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:02:36,816 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-11) [1fca2d5b] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:02:46,884 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-33) [5e994d3f] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:02:46,914 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-33) [5e994d3f] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:02:46,917 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-33) [5e994d3f] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:02:46,933 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-33) [5e994d3f] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7e6ed603 >2012-11-08 10:02:46,957 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-33) [5e994d3f] spmStart polling started: taskId = 94f2d788-cd1a-4085-83a9-64f5fb87afb2 >2012-11-08 10:02:47,998 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-33) [5e994d3f] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:02:47,998 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-33) [5e994d3f] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:02:47,998 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-33) [5e994d3f] spmStart polling ended: taskId = 94f2d788-cd1a-4085-83a9-64f5fb87afb2 task status = finished >2012-11-08 10:02:47,998 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-33) [5e994d3f] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:02:48,025 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-33) [5e994d3f] spmStart polling ended, spm status: Free >2012-11-08 10:02:48,042 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-33) [5e994d3f] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=94f2d788-cd1a-4085-83a9-64f5fb87afb2), log id: 92f06ab >2012-11-08 10:02:48,062 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-33) [5e994d3f] FINISH, HSMClearTaskVDSCommand, log id: 92f06ab >2012-11-08 10:02:48,062 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-33) [5e994d3f] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6e5ec942, log id: 7e6ed603 >2012-11-08 10:02:48,064 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-33) [10ef44aa] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:02:48,087 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-33) [10ef44aa] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:02:58,153 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [3f25781c] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:02:58,182 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [3f25781c] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:02:58,184 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [3f25781c] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:02:58,201 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [3f25781c] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 1b62185 >2012-11-08 10:02:58,231 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [3f25781c] spmStart polling started: taskId = 23634eb1-5411-4b81-9a52-930adbc2c100 >2012-11-08 10:02:59,264 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-85) [3f25781c] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:02:59,264 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-85) [3f25781c] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:02:59,264 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [3f25781c] spmStart polling ended: taskId = 23634eb1-5411-4b81-9a52-930adbc2c100 task status = finished >2012-11-08 10:02:59,264 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [3f25781c] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:02:59,293 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [3f25781c] spmStart polling ended, spm status: Free >2012-11-08 10:02:59,306 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-85) [3f25781c] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=23634eb1-5411-4b81-9a52-930adbc2c100), log id: 32333273 >2012-11-08 10:02:59,335 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-85) [3f25781c] FINISH, HSMClearTaskVDSCommand, log id: 32333273 >2012-11-08 10:02:59,335 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [3f25781c] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@207cb23d, log id: 1b62185 >2012-11-08 10:02:59,337 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-85) [5a99d374] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:02:59,367 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [5a99d374] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:03:06,104 INFO [org.ovirt.engine.core.bll.SetVmTicketCommand] (ajp-/127.0.0.1:8702-5) [62c90a97] Running command: SetVmTicketCommand internal: false. Entities affected : ID: 661d6af9-9024-4744-91b1-258e38466815 Type: VM >2012-11-08 10:03:06,131 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp-/127.0.0.1:8702-5) [62c90a97] START, SetVmTicketVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=661d6af9-9024-4744-91b1-258e38466815, ticket=rerkuoNe4yxr, validTime=120,m userName=admin@internal, userId=fdfc627c-d875-11e0-90f0-83df133b58cc), log id: 55ce1c7 >2012-11-08 10:03:06,161 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp-/127.0.0.1:8702-5) [62c90a97] FINISH, SetVmTicketVDSCommand, log id: 55ce1c7 >2012-11-08 10:03:09,427 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [765f39c1] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:03:09,453 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [765f39c1] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:03:09,456 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [765f39c1] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:03:09,472 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [765f39c1] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2641dd4f >2012-11-08 10:03:09,504 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [765f39c1] spmStart polling started: taskId = 855e3cac-4f8a-4844-b79f-b88e22cd0532 >2012-11-08 10:03:10,541 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-14) [765f39c1] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:03:10,541 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-14) [765f39c1] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:03:10,541 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [765f39c1] spmStart polling ended: taskId = 855e3cac-4f8a-4844-b79f-b88e22cd0532 task status = finished >2012-11-08 10:03:10,541 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [765f39c1] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:03:10,571 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [765f39c1] spmStart polling ended, spm status: Free >2012-11-08 10:03:10,585 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-14) [765f39c1] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=855e3cac-4f8a-4844-b79f-b88e22cd0532), log id: 3339ebe2 >2012-11-08 10:03:10,603 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-14) [765f39c1] FINISH, HSMClearTaskVDSCommand, log id: 3339ebe2 >2012-11-08 10:03:10,603 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [765f39c1] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1fad5cb4, log id: 2641dd4f >2012-11-08 10:03:10,605 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-14) [2ea13dab] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:03:10,628 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [2ea13dab] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:03:20,687 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-54) [543162bd] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:03:20,718 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-54) [543162bd] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:03:20,720 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-54) [543162bd] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:03:20,739 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) [543162bd] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2fdff4a2 >2012-11-08 10:03:20,769 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) [543162bd] spmStart polling started: taskId = cd70c4a6-4a1e-4e48-ad1e-5162a73d7aa8 >2012-11-08 10:03:21,808 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-54) [543162bd] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:03:21,808 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-54) [543162bd] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:03:21,808 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) [543162bd] spmStart polling ended: taskId = cd70c4a6-4a1e-4e48-ad1e-5162a73d7aa8 task status = finished >2012-11-08 10:03:21,808 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) [543162bd] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:03:21,835 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) [543162bd] spmStart polling ended, spm status: Free >2012-11-08 10:03:21,847 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-54) [543162bd] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=cd70c4a6-4a1e-4e48-ad1e-5162a73d7aa8), log id: 1b683f20 >2012-11-08 10:03:21,869 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-54) [543162bd] FINISH, HSMClearTaskVDSCommand, log id: 1b683f20 >2012-11-08 10:03:21,869 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) [543162bd] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@59085bee, log id: 2fdff4a2 >2012-11-08 10:03:21,871 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-54) [33cb93ec] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:03:21,896 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-54) [33cb93ec] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:03:31,962 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [5e18cdc1] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:03:31,995 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [5e18cdc1] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:03:31,998 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [5e18cdc1] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:03:32,016 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-90) [5e18cdc1] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 54b83b8f >2012-11-08 10:03:32,043 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-90) [5e18cdc1] spmStart polling started: taskId = 2fea7657-dae7-4624-9a79-473a9d14d94f >2012-11-08 10:03:33,080 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-90) [5e18cdc1] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:03:33,080 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-90) [5e18cdc1] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:03:33,081 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-90) [5e18cdc1] spmStart polling ended: taskId = 2fea7657-dae7-4624-9a79-473a9d14d94f task status = finished >2012-11-08 10:03:33,081 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-90) [5e18cdc1] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:03:33,105 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-90) [5e18cdc1] spmStart polling ended, spm status: Free >2012-11-08 10:03:33,121 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-90) [5e18cdc1] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=2fea7657-dae7-4624-9a79-473a9d14d94f), log id: 56cf09d6 >2012-11-08 10:03:33,140 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-90) [5e18cdc1] FINISH, HSMClearTaskVDSCommand, log id: 56cf09d6 >2012-11-08 10:03:33,140 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-90) [5e18cdc1] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@328eed20, log id: 54b83b8f >2012-11-08 10:03:33,142 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-90) [4571e9ce] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:03:33,165 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [4571e9ce] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:03:43,223 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-11) [1fca2d5b] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:03:43,250 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-11) [1fca2d5b] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:03:43,253 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-11) [1fca2d5b] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:03:43,269 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-11) [1fca2d5b] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 1f6fca2e >2012-11-08 10:03:43,293 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-11) [1fca2d5b] spmStart polling started: taskId = 53f4ec3e-9334-4a04-bb32-bb03c3ff0854 >2012-11-08 10:03:44,331 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-11) [1fca2d5b] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:03:44,331 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-11) [1fca2d5b] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:03:44,331 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-11) [1fca2d5b] spmStart polling ended: taskId = 53f4ec3e-9334-4a04-bb32-bb03c3ff0854 task status = finished >2012-11-08 10:03:44,331 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-11) [1fca2d5b] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:03:44,363 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-11) [1fca2d5b] spmStart polling ended, spm status: Free >2012-11-08 10:03:44,375 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-11) [1fca2d5b] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=53f4ec3e-9334-4a04-bb32-bb03c3ff0854), log id: 70239b81 >2012-11-08 10:03:44,402 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-11) [1fca2d5b] FINISH, HSMClearTaskVDSCommand, log id: 70239b81 >2012-11-08 10:03:44,402 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-11) [1fca2d5b] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3f56c10c, log id: 1f6fca2e >2012-11-08 10:03:44,404 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-11) [589c1b50] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:03:44,430 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-11) [589c1b50] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:03:54,494 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-49) [41719740] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:03:54,521 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-49) [41719740] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:03:54,524 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-49) [41719740] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:03:54,540 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-49) [41719740] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: ee213da >2012-11-08 10:03:54,563 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-49) [41719740] spmStart polling started: taskId = 28e13058-b548-4951-ac08-39ec9d62e82c >2012-11-08 10:03:55,597 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-49) [41719740] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:03:55,597 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-49) [41719740] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:03:55,597 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-49) [41719740] spmStart polling ended: taskId = 28e13058-b548-4951-ac08-39ec9d62e82c task status = finished >2012-11-08 10:03:55,597 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-49) [41719740] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:03:55,627 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-49) [41719740] spmStart polling ended, spm status: Free >2012-11-08 10:03:55,647 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-49) [41719740] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=28e13058-b548-4951-ac08-39ec9d62e82c), log id: 1b206a40 >2012-11-08 10:03:55,664 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-49) [41719740] FINISH, HSMClearTaskVDSCommand, log id: 1b206a40 >2012-11-08 10:03:55,664 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-49) [41719740] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6d1f9da9, log id: ee213da >2012-11-08 10:03:55,666 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-49) [36a7fa37] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:03:55,688 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-49) [36a7fa37] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:04:05,741 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [4a5fad97] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:04:05,769 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [4a5fad97] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:04:05,771 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [4a5fad97] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:04:05,787 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [4a5fad97] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5798a627 >2012-11-08 10:04:05,811 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [4a5fad97] spmStart polling started: taskId = 41fa3e81-a94e-45db-b9c1-be2dd575c1cb >2012-11-08 10:04:06,846 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-69) [4a5fad97] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:04:06,846 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-69) [4a5fad97] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:04:06,846 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [4a5fad97] spmStart polling ended: taskId = 41fa3e81-a94e-45db-b9c1-be2dd575c1cb task status = finished >2012-11-08 10:04:06,846 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [4a5fad97] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:04:06,878 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [4a5fad97] spmStart polling ended, spm status: Free >2012-11-08 10:04:06,891 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-69) [4a5fad97] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=41fa3e81-a94e-45db-b9c1-be2dd575c1cb), log id: 60d788c7 >2012-11-08 10:04:06,914 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-69) [4a5fad97] FINISH, HSMClearTaskVDSCommand, log id: 60d788c7 >2012-11-08 10:04:06,914 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [4a5fad97] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6654f27c, log id: 5798a627 >2012-11-08 10:04:06,916 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-69) [52fbdf60] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:04:06,932 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [52fbdf60] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:04:16,987 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) [4e8ee24c] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:04:17,016 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) [4e8ee24c] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:04:17,019 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) [4e8ee24c] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:04:17,037 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-51) [4e8ee24c] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4fc1267e >2012-11-08 10:04:17,059 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-51) [4e8ee24c] spmStart polling started: taskId = 4568f393-1d4a-41b5-b8fe-4f69718f8a53 >2012-11-08 10:04:18,090 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-51) [4e8ee24c] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:04:18,090 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-51) [4e8ee24c] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:04:18,091 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-51) [4e8ee24c] spmStart polling ended: taskId = 4568f393-1d4a-41b5-b8fe-4f69718f8a53 task status = finished >2012-11-08 10:04:18,091 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-51) [4e8ee24c] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:04:18,119 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-51) [4e8ee24c] spmStart polling ended, spm status: Free >2012-11-08 10:04:18,137 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-51) [4e8ee24c] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=4568f393-1d4a-41b5-b8fe-4f69718f8a53), log id: 62fcad76 >2012-11-08 10:04:18,154 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-51) [4e8ee24c] FINISH, HSMClearTaskVDSCommand, log id: 62fcad76 >2012-11-08 10:04:18,154 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-51) [4e8ee24c] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3ae050bf, log id: 4fc1267e >2012-11-08 10:04:18,157 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-51) [5cd549cb] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:04:18,181 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) [5cd549cb] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:04:28,240 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-54) [33cb93ec] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:04:28,268 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-54) [33cb93ec] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:04:28,271 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-54) [33cb93ec] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:04:28,287 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) [33cb93ec] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 69fe7f88 >2012-11-08 10:04:28,309 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) [33cb93ec] spmStart polling started: taskId = 4d8f8773-a2fb-4aee-9079-0cdc4bca7c99 >2012-11-08 10:04:29,340 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-54) [33cb93ec] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:04:29,340 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-54) [33cb93ec] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:04:29,340 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) [33cb93ec] spmStart polling ended: taskId = 4d8f8773-a2fb-4aee-9079-0cdc4bca7c99 task status = finished >2012-11-08 10:04:29,340 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) [33cb93ec] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:04:29,368 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) [33cb93ec] spmStart polling ended, spm status: Free >2012-11-08 10:04:29,386 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-54) [33cb93ec] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=4d8f8773-a2fb-4aee-9079-0cdc4bca7c99), log id: 71174f8e >2012-11-08 10:04:29,412 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-54) [33cb93ec] FINISH, HSMClearTaskVDSCommand, log id: 71174f8e >2012-11-08 10:04:29,412 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) [33cb93ec] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@31682da3, log id: 69fe7f88 >2012-11-08 10:04:29,413 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-54) [2e8c41ac] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:04:29,442 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-54) [2e8c41ac] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:04:39,509 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) [447d86d2] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:04:39,533 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) [447d86d2] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:04:39,536 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) [447d86d2] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:04:39,556 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-86) [447d86d2] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 21fda050 >2012-11-08 10:04:39,591 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-86) [447d86d2] spmStart polling started: taskId = 8009141f-80e5-42af-8334-202ad17c1495 >2012-11-08 10:04:40,630 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-86) [447d86d2] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:04:40,630 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-86) [447d86d2] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:04:40,630 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-86) [447d86d2] spmStart polling ended: taskId = 8009141f-80e5-42af-8334-202ad17c1495 task status = finished >2012-11-08 10:04:40,630 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-86) [447d86d2] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:04:40,659 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-86) [447d86d2] spmStart polling ended, spm status: Free >2012-11-08 10:04:40,671 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-86) [447d86d2] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=8009141f-80e5-42af-8334-202ad17c1495), log id: 22ac7675 >2012-11-08 10:04:40,694 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-86) [447d86d2] FINISH, HSMClearTaskVDSCommand, log id: 22ac7675 >2012-11-08 10:04:40,694 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-86) [447d86d2] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@68950d1b, log id: 21fda050 >2012-11-08 10:04:40,695 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-86) [118f9d41] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:04:40,720 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) [118f9d41] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:04:50,780 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-20) [79d6527d] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:04:50,807 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-20) [79d6527d] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:04:50,810 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-20) [79d6527d] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:04:50,826 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-20) [79d6527d] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 70d0dda7 >2012-11-08 10:04:50,852 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-20) [79d6527d] spmStart polling started: taskId = 902893e0-e264-445a-a25a-b2df2af69f20 >2012-11-08 10:04:51,884 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-20) [79d6527d] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:04:51,884 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-20) [79d6527d] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:04:51,884 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-20) [79d6527d] spmStart polling ended: taskId = 902893e0-e264-445a-a25a-b2df2af69f20 task status = finished >2012-11-08 10:04:51,884 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-20) [79d6527d] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:04:51,915 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-20) [79d6527d] spmStart polling ended, spm status: Free >2012-11-08 10:04:51,929 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-20) [79d6527d] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=902893e0-e264-445a-a25a-b2df2af69f20), log id: 48559eee >2012-11-08 10:04:51,952 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-20) [79d6527d] FINISH, HSMClearTaskVDSCommand, log id: 48559eee >2012-11-08 10:04:51,952 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-20) [79d6527d] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6d35405b, log id: 70d0dda7 >2012-11-08 10:04:51,954 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-20) [43144e60] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:04:51,971 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-20) [43144e60] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:05:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-55) [367e311c] Autorecovering hosts is disabled, skipping >2012-11-08 10:05:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-55) [367e311c] Autorecovering storage domains is disabled, skipping >2012-11-08 10:05:02,027 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [7e088618] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:05:02,049 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [7e088618] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:05:02,052 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [7e088618] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:05:02,067 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [7e088618] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 63723df2 >2012-11-08 10:05:02,090 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [7e088618] spmStart polling started: taskId = 9a0eed5e-81aa-40fa-b317-f2720e8c6e5c >2012-11-08 10:05:03,119 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-94) [7e088618] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:05:03,119 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-94) [7e088618] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:05:03,120 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [7e088618] spmStart polling ended: taskId = 9a0eed5e-81aa-40fa-b317-f2720e8c6e5c task status = finished >2012-11-08 10:05:03,120 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [7e088618] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:05:03,147 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [7e088618] spmStart polling ended, spm status: Free >2012-11-08 10:05:03,165 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-94) [7e088618] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=9a0eed5e-81aa-40fa-b317-f2720e8c6e5c), log id: bc789f8 >2012-11-08 10:05:03,181 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-94) [7e088618] FINISH, HSMClearTaskVDSCommand, log id: bc789f8 >2012-11-08 10:05:03,181 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [7e088618] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@5932fed2, log id: 63723df2 >2012-11-08 10:05:03,183 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-94) [22f7babe] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:05:03,209 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [22f7babe] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:05:13,266 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-81) [2d6e717] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:05:13,294 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-81) [2d6e717] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:05:13,297 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-81) [2d6e717] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:05:13,314 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-81) [2d6e717] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7c27612f >2012-11-08 10:05:13,344 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-81) [2d6e717] spmStart polling started: taskId = f4c14ad5-6f2c-40d9-95f6-95d4f0c1dc5c >2012-11-08 10:05:14,390 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-81) [2d6e717] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:05:14,390 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-81) [2d6e717] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:05:14,391 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-81) [2d6e717] spmStart polling ended: taskId = f4c14ad5-6f2c-40d9-95f6-95d4f0c1dc5c task status = finished >2012-11-08 10:05:14,391 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-81) [2d6e717] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:05:14,418 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-81) [2d6e717] spmStart polling ended, spm status: Free >2012-11-08 10:05:14,436 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-81) [2d6e717] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=f4c14ad5-6f2c-40d9-95f6-95d4f0c1dc5c), log id: 640a085e >2012-11-08 10:05:14,460 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-81) [2d6e717] FINISH, HSMClearTaskVDSCommand, log id: 640a085e >2012-11-08 10:05:14,460 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-81) [2d6e717] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@fe33b31, log id: 7c27612f >2012-11-08 10:05:14,462 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-81) [f43161d] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:05:14,478 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-81) [f43161d] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:05:24,543 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [4174cf5b] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:05:24,570 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [4174cf5b] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:05:24,573 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [4174cf5b] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:05:24,592 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [4174cf5b] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5e60d01a >2012-11-08 10:05:24,618 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [4174cf5b] spmStart polling started: taskId = 5291de0b-78f9-4584-9117-e1dce66c61fd >2012-11-08 10:05:25,653 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-7) [4174cf5b] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:05:25,653 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-7) [4174cf5b] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:05:25,653 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [4174cf5b] spmStart polling ended: taskId = 5291de0b-78f9-4584-9117-e1dce66c61fd task status = finished >2012-11-08 10:05:25,654 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [4174cf5b] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:05:25,686 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [4174cf5b] spmStart polling ended, spm status: Free >2012-11-08 10:05:25,700 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-7) [4174cf5b] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=5291de0b-78f9-4584-9117-e1dce66c61fd), log id: 7b2eb41f >2012-11-08 10:05:25,722 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-7) [4174cf5b] FINISH, HSMClearTaskVDSCommand, log id: 7b2eb41f >2012-11-08 10:05:25,722 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [4174cf5b] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@37339e67, log id: 5e60d01a >2012-11-08 10:05:25,724 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-7) [1de49881] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:05:25,748 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [1de49881] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:05:35,802 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-54) [2e8c41ac] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:05:35,833 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-54) [2e8c41ac] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:05:35,836 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-54) [2e8c41ac] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:05:35,854 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) [2e8c41ac] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 346f34e7 >2012-11-08 10:05:35,876 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) [2e8c41ac] spmStart polling started: taskId = 4346e191-bd04-4171-b784-a5b7a4a8014c >2012-11-08 10:05:36,908 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-54) [2e8c41ac] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:05:36,908 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-54) [2e8c41ac] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:05:36,908 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) [2e8c41ac] spmStart polling ended: taskId = 4346e191-bd04-4171-b784-a5b7a4a8014c task status = finished >2012-11-08 10:05:36,908 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) [2e8c41ac] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:05:36,936 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) [2e8c41ac] spmStart polling ended, spm status: Free >2012-11-08 10:05:36,953 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-54) [2e8c41ac] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=4346e191-bd04-4171-b784-a5b7a4a8014c), log id: 65fc0d37 >2012-11-08 10:05:36,970 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-54) [2e8c41ac] FINISH, HSMClearTaskVDSCommand, log id: 65fc0d37 >2012-11-08 10:05:36,970 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-54) [2e8c41ac] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6c7ca30, log id: 346f34e7 >2012-11-08 10:05:36,972 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-54) [26ab18ca] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:05:36,992 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-54) [26ab18ca] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:05:43,700 INFO [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (ajp-/127.0.0.1:8702-16) [4ebe36e1] Lock Acquired to object EngineLock [exclusiveLocks= key: disrep6 value: VM_NAME >, sharedLocks= ] >2012-11-08 10:05:43,715 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-16) [4ebe36e1] START, IsValidVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 2c1e29b >2012-11-08 10:05:43,716 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-16) [4ebe36e1] FINISH, IsValidVDSCommand, return: false, log id: 2c1e29b >2012-11-08 10:05:43,716 WARN [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (ajp-/127.0.0.1:8702-16) [4ebe36e1] CanDoAction of action AddVmFromScratch failed. Reasons:VAR__ACTION__ADD,VAR__TYPE__VM,IMAGE_REPOSITORY_NOT_FOUND >2012-11-08 10:05:43,716 INFO [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (ajp-/127.0.0.1:8702-16) [4ebe36e1] Lock freed to object EngineLock [exclusiveLocks= key: disrep6 value: VM_NAME >, sharedLocks= ] >2012-11-08 10:05:47,044 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [52fbdf60] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:05:47,074 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [52fbdf60] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:05:47,077 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [52fbdf60] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:05:47,093 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [52fbdf60] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6465bd7b >2012-11-08 10:05:47,127 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [52fbdf60] spmStart polling started: taskId = 340ba13f-04e6-49a1-a175-52914945e7f6 >2012-11-08 10:05:48,161 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-69) [52fbdf60] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:05:48,161 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-69) [52fbdf60] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:05:48,161 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [52fbdf60] spmStart polling ended: taskId = 340ba13f-04e6-49a1-a175-52914945e7f6 task status = finished >2012-11-08 10:05:48,161 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [52fbdf60] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:05:48,193 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [52fbdf60] spmStart polling ended, spm status: Free >2012-11-08 10:05:48,206 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-69) [52fbdf60] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=340ba13f-04e6-49a1-a175-52914945e7f6), log id: 15481783 >2012-11-08 10:05:48,224 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-69) [52fbdf60] FINISH, HSMClearTaskVDSCommand, log id: 15481783 >2012-11-08 10:05:48,224 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [52fbdf60] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2fffa45f, log id: 6465bd7b >2012-11-08 10:05:48,226 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-69) [38807197] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:05:48,248 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [38807197] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:05:56,227 INFO [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (ajp-/127.0.0.1:8702-1) [6fe72394] Lock Acquired to object EngineLock [exclusiveLocks= key: disrep6 value: VM_NAME >, sharedLocks= ] >2012-11-08 10:05:56,241 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-1) [6fe72394] START, IsValidVDSCommand( storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 6004d388 >2012-11-08 10:05:56,242 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (ajp-/127.0.0.1:8702-1) [6fe72394] FINISH, IsValidVDSCommand, return: false, log id: 6004d388 >2012-11-08 10:05:56,242 WARN [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (ajp-/127.0.0.1:8702-1) [6fe72394] CanDoAction of action AddVmFromScratch failed. Reasons:VAR__ACTION__ADD,VAR__TYPE__VM,IMAGE_REPOSITORY_NOT_FOUND >2012-11-08 10:05:56,242 INFO [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (ajp-/127.0.0.1:8702-1) [6fe72394] Lock freed to object EngineLock [exclusiveLocks= key: disrep6 value: VM_NAME >, sharedLocks= ] >2012-11-08 10:05:58,307 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-92) hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:05:58,335 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-92) SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:05:58,338 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-92) starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:05:58,355 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-92) START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5071b3a6 >2012-11-08 10:05:58,378 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-92) spmStart polling started: taskId = 274d06e0-061e-4052-9bb8-415bf80ea0d1 >2012-11-08 10:05:59,421 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-92) Failed in HSMGetTaskStatusVDS method >2012-11-08 10:05:59,421 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-92) Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:05:59,421 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-92) spmStart polling ended: taskId = 274d06e0-061e-4052-9bb8-415bf80ea0d1 task status = finished >2012-11-08 10:05:59,421 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-92) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:05:59,453 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-92) spmStart polling ended, spm status: Free >2012-11-08 10:05:59,471 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-92) START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=274d06e0-061e-4052-9bb8-415bf80ea0d1), log id: 786c4e4a >2012-11-08 10:05:59,489 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-92) FINISH, HSMClearTaskVDSCommand, log id: 786c4e4a >2012-11-08 10:05:59,489 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-92) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@e881a0a, log id: 5071b3a6 >2012-11-08 10:05:59,491 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-92) [9658dfa] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:05:59,514 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-92) [9658dfa] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:06:09,572 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [501c9b34] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:06:09,601 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [501c9b34] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:06:09,604 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [501c9b34] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:06:09,622 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-50) [501c9b34] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 388f1e58 >2012-11-08 10:06:09,652 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-50) [501c9b34] spmStart polling started: taskId = 05524ee2-cae4-4f6e-92b4-5aac56bc963e >2012-11-08 10:06:10,686 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-50) [501c9b34] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:06:10,686 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-50) [501c9b34] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:06:10,686 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-50) [501c9b34] spmStart polling ended: taskId = 05524ee2-cae4-4f6e-92b4-5aac56bc963e task status = finished >2012-11-08 10:06:10,686 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-50) [501c9b34] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:06:10,716 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-50) [501c9b34] spmStart polling ended, spm status: Free >2012-11-08 10:06:10,727 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-50) [501c9b34] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=05524ee2-cae4-4f6e-92b4-5aac56bc963e), log id: 5409ff0f >2012-11-08 10:06:10,750 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-50) [501c9b34] FINISH, HSMClearTaskVDSCommand, log id: 5409ff0f >2012-11-08 10:06:10,750 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-50) [501c9b34] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@5c485ce3, log id: 388f1e58 >2012-11-08 10:06:10,752 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-50) [12ca6a6a] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:06:10,771 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [12ca6a6a] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:06:11,453 INFO [org.ovirt.engine.core.bll.SetVmTicketCommand] (ajp-/127.0.0.1:8702-15) [1844896d] Running command: SetVmTicketCommand internal: false. Entities affected : ID: 661d6af9-9024-4744-91b1-258e38466815 Type: VM >2012-11-08 10:06:11,477 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp-/127.0.0.1:8702-15) [1844896d] START, SetVmTicketVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, vmId=661d6af9-9024-4744-91b1-258e38466815, ticket=t525RyYR3R0b, validTime=120,m userName=admin@internal, userId=fdfc627c-d875-11e0-90f0-83df133b58cc), log id: 46150908 >2012-11-08 10:06:11,509 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp-/127.0.0.1:8702-15) [1844896d] FINISH, SetVmTicketVDSCommand, log id: 46150908 >2012-11-08 10:06:20,836 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-62) [7d0da67e] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:06:20,865 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-62) [7d0da67e] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:06:20,870 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-62) [7d0da67e] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:06:20,884 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-62) [7d0da67e] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 1bbc5e29 >2012-11-08 10:06:20,907 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-62) [7d0da67e] spmStart polling started: taskId = 6e31cea2-53de-420d-a928-e066d58e104a >2012-11-08 10:06:21,944 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-62) [7d0da67e] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:06:21,944 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-62) [7d0da67e] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:06:21,944 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-62) [7d0da67e] spmStart polling ended: taskId = 6e31cea2-53de-420d-a928-e066d58e104a task status = finished >2012-11-08 10:06:21,944 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-62) [7d0da67e] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:06:21,977 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-62) [7d0da67e] spmStart polling ended, spm status: Free >2012-11-08 10:06:21,993 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-62) [7d0da67e] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=6e31cea2-53de-420d-a928-e066d58e104a), log id: 454378b5 >2012-11-08 10:06:22,010 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-62) [7d0da67e] FINISH, HSMClearTaskVDSCommand, log id: 454378b5 >2012-11-08 10:06:22,010 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-62) [7d0da67e] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@50a9d699, log id: 1bbc5e29 >2012-11-08 10:06:22,012 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-62) [3e48002e] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:06:22,031 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-62) [3e48002e] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:06:32,082 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [6f1c8a8f] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:06:32,111 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [6f1c8a8f] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:06:32,114 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [6f1c8a8f] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:06:32,131 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-27) [6f1c8a8f] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 34f1664 >2012-11-08 10:06:32,153 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-27) [6f1c8a8f] spmStart polling started: taskId = 96f8a330-a3ce-4dfc-a219-5ab230b011d0 >2012-11-08 10:06:33,184 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-27) [6f1c8a8f] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:06:33,184 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-27) [6f1c8a8f] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:06:33,184 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-27) [6f1c8a8f] spmStart polling ended: taskId = 96f8a330-a3ce-4dfc-a219-5ab230b011d0 task status = finished >2012-11-08 10:06:33,185 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-27) [6f1c8a8f] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:06:33,213 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-27) [6f1c8a8f] spmStart polling ended, spm status: Free >2012-11-08 10:06:33,230 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-27) [6f1c8a8f] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=96f8a330-a3ce-4dfc-a219-5ab230b011d0), log id: 7e45f407 >2012-11-08 10:06:33,247 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-27) [6f1c8a8f] FINISH, HSMClearTaskVDSCommand, log id: 7e45f407 >2012-11-08 10:06:33,247 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-27) [6f1c8a8f] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@22983855, log id: 34f1664 >2012-11-08 10:06:33,250 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-27) [379a6631] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:06:33,273 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [379a6631] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:06:37,976 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-36) [5e4b1d15] starting ProcessDomainRecovery for domain e0cce407-ebea-4322-b47e-7aceb73a7a6f:DISREP >2012-11-08 10:06:38,008 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-36) [5e4b1d15] domain e0cce407-ebea-4322-b47e-7aceb73a7a6f:DISREP was reported by all hosts in status UP as problematic. Moving the Domain to NonOperational. >2012-11-08 10:06:38,020 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (QuartzScheduler_Worker-36) [14862cb3] Lock Acquired to object EngineLock [exclusiveLocks= key: e0cce407-ebea-4322-b47e-7aceb73a7a6f value: STORAGE >, sharedLocks= key: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 value: POOL >] >2012-11-08 10:06:38,020 WARN [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (QuartzScheduler_Worker-36) [14862cb3] CanDoAction of action DeactivateStorageDomain failed. Reasons:VAR__TYPE__STORAGE__DOMAIN,VAR__ACTION__DEACTIVATE,ACTION_TYPE_FAILED_STORAGE_DOMAIN_STATUS_ILLEGAL >2012-11-08 10:06:38,020 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (QuartzScheduler_Worker-36) [14862cb3] Lock freed to object EngineLock [exclusiveLocks= key: e0cce407-ebea-4322-b47e-7aceb73a7a6f value: STORAGE >, sharedLocks= key: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 value: POOL >] >2012-11-08 10:06:43,321 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-56) [63abd9c7] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:06:43,350 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-56) [63abd9c7] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:06:43,355 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-56) [63abd9c7] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:06:43,372 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-56) [63abd9c7] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3e8e6d5c >2012-11-08 10:06:43,400 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-56) [63abd9c7] spmStart polling started: taskId = 439fda29-0a03-4c7c-a867-cc8a17e79b0d >2012-11-08 10:06:44,438 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-56) [63abd9c7] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:06:44,438 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-56) [63abd9c7] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:06:44,438 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-56) [63abd9c7] spmStart polling ended: taskId = 439fda29-0a03-4c7c-a867-cc8a17e79b0d task status = finished >2012-11-08 10:06:44,438 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-56) [63abd9c7] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:06:44,470 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-56) [63abd9c7] spmStart polling ended, spm status: Free >2012-11-08 10:06:44,484 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-56) [63abd9c7] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=439fda29-0a03-4c7c-a867-cc8a17e79b0d), log id: 37501f1 >2012-11-08 10:06:44,507 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-56) [63abd9c7] FINISH, HSMClearTaskVDSCommand, log id: 37501f1 >2012-11-08 10:06:44,507 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-56) [63abd9c7] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@76d276fc, log id: 3e8e6d5c >2012-11-08 10:06:44,509 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-56) [7381d755] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:06:44,531 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-56) [7381d755] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:06:47,190 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-76) [2879b44b] domain e0cce407-ebea-4322-b47e-7aceb73a7a6f:DISREP in problem. vds: rhs-gp-srv11.lab.eng.blr.redhat.com >2012-11-08 10:06:54,594 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [38807197] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:06:54,622 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [38807197] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:06:54,625 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [38807197] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:06:54,639 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [38807197] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4ea24ff8 >2012-11-08 10:06:54,662 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [38807197] spmStart polling started: taskId = 8b0dffb6-5377-469e-9eb8-5da72e850853 >2012-11-08 10:06:55,698 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-69) [38807197] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:06:55,698 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-69) [38807197] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:06:55,698 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [38807197] spmStart polling ended: taskId = 8b0dffb6-5377-469e-9eb8-5da72e850853 task status = finished >2012-11-08 10:06:55,698 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [38807197] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:06:55,730 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [38807197] spmStart polling ended, spm status: Free >2012-11-08 10:06:55,749 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-69) [38807197] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=8b0dffb6-5377-469e-9eb8-5da72e850853), log id: 675dadc6 >2012-11-08 10:06:55,766 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-69) [38807197] FINISH, HSMClearTaskVDSCommand, log id: 675dadc6 >2012-11-08 10:06:55,766 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [38807197] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@373bd72e, log id: 4ea24ff8 >2012-11-08 10:06:55,770 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-69) [39ac2956] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:06:55,794 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [39ac2956] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:07:05,852 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-92) [9658dfa] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:07:05,883 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-92) [9658dfa] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:07:05,884 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-92) [9658dfa] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:07:05,896 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-92) [9658dfa] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 46523ccc >2012-11-08 10:07:05,921 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-92) [9658dfa] spmStart polling started: taskId = 259d1d09-f172-49fb-b920-1cac2d3b07da >2012-11-08 10:07:06,966 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-92) [9658dfa] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:07:06,966 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-92) [9658dfa] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:07:06,966 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-92) [9658dfa] spmStart polling ended: taskId = 259d1d09-f172-49fb-b920-1cac2d3b07da task status = finished >2012-11-08 10:07:06,966 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-92) [9658dfa] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:07:06,996 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-92) [9658dfa] spmStart polling ended, spm status: Free >2012-11-08 10:07:07,010 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-92) [9658dfa] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=259d1d09-f172-49fb-b920-1cac2d3b07da), log id: 6875c828 >2012-11-08 10:07:07,032 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-92) [9658dfa] FINISH, HSMClearTaskVDSCommand, log id: 6875c828 >2012-11-08 10:07:07,032 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-92) [9658dfa] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2b80531f, log id: 46523ccc >2012-11-08 10:07:07,034 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-92) [75d12e88] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:07:07,052 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-92) [75d12e88] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:07:17,103 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [12ca6a6a] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:07:17,130 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [12ca6a6a] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:07:17,133 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [12ca6a6a] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:07:17,150 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-50) [12ca6a6a] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2d58d491 >2012-11-08 10:07:17,181 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-50) [12ca6a6a] spmStart polling started: taskId = 1f201f99-7e21-471e-a777-bf729a9aafef >2012-11-08 10:07:18,217 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-50) [12ca6a6a] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:07:18,217 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-50) [12ca6a6a] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:07:18,217 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-50) [12ca6a6a] spmStart polling ended: taskId = 1f201f99-7e21-471e-a777-bf729a9aafef task status = finished >2012-11-08 10:07:18,217 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-50) [12ca6a6a] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:07:18,244 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-50) [12ca6a6a] spmStart polling ended, spm status: Free >2012-11-08 10:07:18,256 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-50) [12ca6a6a] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=1f201f99-7e21-471e-a777-bf729a9aafef), log id: 9cf8ff4 >2012-11-08 10:07:18,278 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-50) [12ca6a6a] FINISH, HSMClearTaskVDSCommand, log id: 9cf8ff4 >2012-11-08 10:07:18,278 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-50) [12ca6a6a] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@19f3e134, log id: 2d58d491 >2012-11-08 10:07:18,280 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-50) [5a462a3b] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:07:18,303 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [5a462a3b] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:07:28,375 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [39ac2956] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:07:28,404 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [39ac2956] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:07:28,407 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [39ac2956] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:07:28,421 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [39ac2956] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 588fbfb6 >2012-11-08 10:07:28,446 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [39ac2956] spmStart polling started: taskId = d96a6d57-3af3-49c8-af50-38e39a9c7b2d >2012-11-08 10:07:29,481 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-69) [39ac2956] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:07:29,481 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-69) [39ac2956] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:07:29,481 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [39ac2956] spmStart polling ended: taskId = d96a6d57-3af3-49c8-af50-38e39a9c7b2d task status = finished >2012-11-08 10:07:29,481 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [39ac2956] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:07:29,509 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [39ac2956] spmStart polling ended, spm status: Free >2012-11-08 10:07:29,529 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-69) [39ac2956] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=d96a6d57-3af3-49c8-af50-38e39a9c7b2d), log id: 27e51fa6 >2012-11-08 10:07:29,555 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-69) [39ac2956] FINISH, HSMClearTaskVDSCommand, log id: 27e51fa6 >2012-11-08 10:07:29,555 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [39ac2956] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3380dfc0, log id: 588fbfb6 >2012-11-08 10:07:29,557 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-69) [51e1d0aa] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:07:29,581 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [51e1d0aa] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:07:39,637 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [379a6631] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:07:39,667 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [379a6631] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:07:39,670 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [379a6631] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:07:39,687 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-27) [379a6631] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6e09743f >2012-11-08 10:07:39,710 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-27) [379a6631] spmStart polling started: taskId = a2db7fa6-5db1-4cbf-a4c6-53a585f60bfb >2012-11-08 10:07:40,745 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-27) [379a6631] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:07:40,745 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-27) [379a6631] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:07:40,746 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-27) [379a6631] spmStart polling ended: taskId = a2db7fa6-5db1-4cbf-a4c6-53a585f60bfb task status = finished >2012-11-08 10:07:40,746 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-27) [379a6631] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:07:40,778 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-27) [379a6631] spmStart polling ended, spm status: Free >2012-11-08 10:07:40,791 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-27) [379a6631] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=a2db7fa6-5db1-4cbf-a4c6-53a585f60bfb), log id: 4577ae19 >2012-11-08 10:07:40,809 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-27) [379a6631] FINISH, HSMClearTaskVDSCommand, log id: 4577ae19 >2012-11-08 10:07:40,809 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-27) [379a6631] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@34470ef0, log id: 6e09743f >2012-11-08 10:07:40,811 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-27) [2525c960] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:07:40,832 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [2525c960] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:07:50,878 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-56) [7381d755] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:07:50,907 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-56) [7381d755] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:07:50,910 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-56) [7381d755] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:07:50,928 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-56) [7381d755] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3a7f22ea >2012-11-08 10:07:50,949 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-56) [7381d755] spmStart polling started: taskId = 69a78989-7d41-4d8f-ac8d-3a72c406375e >2012-11-08 10:07:51,986 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-56) [7381d755] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:07:51,986 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-56) [7381d755] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:07:51,986 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-56) [7381d755] spmStart polling ended: taskId = 69a78989-7d41-4d8f-ac8d-3a72c406375e task status = finished >2012-11-08 10:07:51,986 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-56) [7381d755] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:07:52,017 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-56) [7381d755] spmStart polling ended, spm status: Free >2012-11-08 10:07:52,036 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-56) [7381d755] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=69a78989-7d41-4d8f-ac8d-3a72c406375e), log id: 185a7678 >2012-11-08 10:07:52,052 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-56) [7381d755] FINISH, HSMClearTaskVDSCommand, log id: 185a7678 >2012-11-08 10:07:52,052 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-56) [7381d755] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@5b17ccc5, log id: 3a7f22ea >2012-11-08 10:07:52,055 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-56) [7784c83b] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:07:52,081 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-56) [7784c83b] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:08:02,139 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [431362fc] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:08:02,167 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [431362fc] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:08:02,170 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [431362fc] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:08:02,187 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [431362fc] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7c33283c >2012-11-08 10:08:02,211 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [431362fc] spmStart polling started: taskId = b8e81e24-23e9-4270-96d9-9ee5446b570c >2012-11-08 10:08:03,242 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-5) [431362fc] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:08:03,242 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-5) [431362fc] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:08:03,242 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [431362fc] spmStart polling ended: taskId = b8e81e24-23e9-4270-96d9-9ee5446b570c task status = finished >2012-11-08 10:08:03,242 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [431362fc] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:08:03,269 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [431362fc] spmStart polling ended, spm status: Free >2012-11-08 10:08:03,287 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-5) [431362fc] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=b8e81e24-23e9-4270-96d9-9ee5446b570c), log id: 2c6b79db >2012-11-08 10:08:03,305 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-5) [431362fc] FINISH, HSMClearTaskVDSCommand, log id: 2c6b79db >2012-11-08 10:08:03,305 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [431362fc] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@440d2fa5, log id: 7c33283c >2012-11-08 10:08:03,307 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-5) [3db31c8a] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:08:03,330 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [3db31c8a] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:08:13,378 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [322eb63e] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:08:13,407 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [322eb63e] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:08:13,410 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [322eb63e] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:08:13,426 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [322eb63e] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2d02a45 >2012-11-08 10:08:13,451 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [322eb63e] spmStart polling started: taskId = bb809e88-240a-4c85-88fc-a0c6c837b2f8 >2012-11-08 10:08:14,493 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-25) [322eb63e] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:08:14,493 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-25) [322eb63e] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:08:14,493 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [322eb63e] spmStart polling ended: taskId = bb809e88-240a-4c85-88fc-a0c6c837b2f8 task status = finished >2012-11-08 10:08:14,493 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [322eb63e] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:08:14,520 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [322eb63e] spmStart polling ended, spm status: Free >2012-11-08 10:08:14,536 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-25) [322eb63e] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=bb809e88-240a-4c85-88fc-a0c6c837b2f8), log id: 30e71230 >2012-11-08 10:08:14,554 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-25) [322eb63e] FINISH, HSMClearTaskVDSCommand, log id: 30e71230 >2012-11-08 10:08:14,554 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [322eb63e] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@125b2665, log id: 2d02a45 >2012-11-08 10:08:14,556 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-25) [1e35ba4d] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:08:14,580 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [1e35ba4d] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:08:24,629 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [65296545] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:08:24,652 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [65296545] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:08:24,654 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [65296545] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:08:24,664 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [65296545] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2d3b9f3 >2012-11-08 10:08:24,686 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [65296545] spmStart polling started: taskId = 72b9a6e9-02b8-453a-85ea-1999dab26f6e >2012-11-08 10:08:25,724 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-66) [65296545] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:08:25,724 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-66) [65296545] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:08:25,724 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [65296545] spmStart polling ended: taskId = 72b9a6e9-02b8-453a-85ea-1999dab26f6e task status = finished >2012-11-08 10:08:25,724 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [65296545] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:08:25,756 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [65296545] spmStart polling ended, spm status: Free >2012-11-08 10:08:25,769 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-66) [65296545] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=72b9a6e9-02b8-453a-85ea-1999dab26f6e), log id: 2633b4b7 >2012-11-08 10:08:25,787 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-66) [65296545] FINISH, HSMClearTaskVDSCommand, log id: 2633b4b7 >2012-11-08 10:08:25,787 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [65296545] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@c9d6df9, log id: 2d3b9f3 >2012-11-08 10:08:25,789 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-66) [339efce7] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:08:25,808 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [339efce7] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:08:35,869 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [3395371b] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:08:35,897 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [3395371b] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:08:35,900 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [3395371b] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:08:35,916 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-1) [3395371b] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 1ad2c2a0 >2012-11-08 10:08:35,945 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-1) [3395371b] spmStart polling started: taskId = 4d01c99b-bfbb-4fe3-adc4-506913fb7165 >2012-11-08 10:08:36,978 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-1) [3395371b] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:08:36,979 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-1) [3395371b] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:08:36,979 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-1) [3395371b] spmStart polling ended: taskId = 4d01c99b-bfbb-4fe3-adc4-506913fb7165 task status = finished >2012-11-08 10:08:36,979 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-1) [3395371b] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:08:37,007 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-1) [3395371b] spmStart polling ended, spm status: Free >2012-11-08 10:08:37,020 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-1) [3395371b] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=4d01c99b-bfbb-4fe3-adc4-506913fb7165), log id: 6518becf >2012-11-08 10:08:37,038 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-1) [3395371b] FINISH, HSMClearTaskVDSCommand, log id: 6518becf >2012-11-08 10:08:37,038 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-1) [3395371b] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3b973be3, log id: 1ad2c2a0 >2012-11-08 10:08:37,040 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-1) [45dcd2a5] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:08:37,062 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [45dcd2a5] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:08:47,123 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-92) [75d12e88] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:08:47,148 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-92) [75d12e88] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:08:47,150 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-92) [75d12e88] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:08:47,166 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-92) [75d12e88] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4a82ffba >2012-11-08 10:08:47,189 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-92) [75d12e88] spmStart polling started: taskId = 5711f1a1-958a-4681-8a42-10d232024368 >2012-11-08 10:08:48,225 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-92) [75d12e88] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:08:48,225 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-92) [75d12e88] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:08:48,225 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-92) [75d12e88] spmStart polling ended: taskId = 5711f1a1-958a-4681-8a42-10d232024368 task status = finished >2012-11-08 10:08:48,225 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-92) [75d12e88] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:08:48,256 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-92) [75d12e88] spmStart polling ended, spm status: Free >2012-11-08 10:08:48,274 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-92) [75d12e88] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=5711f1a1-958a-4681-8a42-10d232024368), log id: 56ce0ff4 >2012-11-08 10:08:48,291 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-92) [75d12e88] FINISH, HSMClearTaskVDSCommand, log id: 56ce0ff4 >2012-11-08 10:08:48,291 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-92) [75d12e88] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@727988b1, log id: 4a82ffba >2012-11-08 10:08:48,293 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-92) [8fd572e] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:08:48,317 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-92) [8fd572e] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:08:58,376 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [339efce7] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:08:58,408 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [339efce7] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:08:58,411 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [339efce7] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:08:58,426 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [339efce7] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 494c2429 >2012-11-08 10:08:58,450 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [339efce7] spmStart polling started: taskId = 6cc0ee6d-074d-4a06-947f-abe9e48758df >2012-11-08 10:08:59,491 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-66) [339efce7] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:08:59,491 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-66) [339efce7] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:08:59,491 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [339efce7] spmStart polling ended: taskId = 6cc0ee6d-074d-4a06-947f-abe9e48758df task status = finished >2012-11-08 10:08:59,491 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [339efce7] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:08:59,518 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [339efce7] spmStart polling ended, spm status: Free >2012-11-08 10:08:59,530 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-66) [339efce7] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=6cc0ee6d-074d-4a06-947f-abe9e48758df), log id: 4c518963 >2012-11-08 10:08:59,548 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-66) [339efce7] FINISH, HSMClearTaskVDSCommand, log id: 4c518963 >2012-11-08 10:08:59,548 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [339efce7] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7d12e9b9, log id: 494c2429 >2012-11-08 10:08:59,549 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-66) [64aeb0d5] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:08:59,572 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [64aeb0d5] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:09:09,629 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [1d088d5b] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:09:09,655 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [1d088d5b] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:09:09,658 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [1d088d5b] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:09:09,675 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-97) [1d088d5b] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 57b80928 >2012-11-08 10:09:09,704 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-97) [1d088d5b] spmStart polling started: taskId = c9f2ff58-3c68-4d21-9da8-b0eec96e4579 >2012-11-08 10:09:10,736 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-97) [1d088d5b] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:09:10,737 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-97) [1d088d5b] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:09:10,737 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-97) [1d088d5b] spmStart polling ended: taskId = c9f2ff58-3c68-4d21-9da8-b0eec96e4579 task status = finished >2012-11-08 10:09:10,737 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-97) [1d088d5b] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:09:10,765 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-97) [1d088d5b] spmStart polling ended, spm status: Free >2012-11-08 10:09:10,779 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-97) [1d088d5b] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=c9f2ff58-3c68-4d21-9da8-b0eec96e4579), log id: 5d4084f5 >2012-11-08 10:09:10,797 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-97) [1d088d5b] FINISH, HSMClearTaskVDSCommand, log id: 5d4084f5 >2012-11-08 10:09:10,797 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-97) [1d088d5b] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@36d920a7, log id: 57b80928 >2012-11-08 10:09:10,799 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-97) [68eed012] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:09:10,825 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [68eed012] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:09:20,889 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [2525c960] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:09:20,915 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [2525c960] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:09:20,918 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [2525c960] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:09:20,934 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-27) [2525c960] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3470074e >2012-11-08 10:09:20,958 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-27) [2525c960] spmStart polling started: taskId = acdd18b1-f595-4b4f-97e0-b8efccc2132f >2012-11-08 10:09:21,987 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-27) [2525c960] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:09:21,987 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-27) [2525c960] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:09:21,987 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-27) [2525c960] spmStart polling ended: taskId = acdd18b1-f595-4b4f-97e0-b8efccc2132f task status = finished >2012-11-08 10:09:21,987 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-27) [2525c960] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:09:22,016 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-27) [2525c960] spmStart polling ended, spm status: Free >2012-11-08 10:09:22,033 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-27) [2525c960] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=acdd18b1-f595-4b4f-97e0-b8efccc2132f), log id: 61aabe18 >2012-11-08 10:09:22,051 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-27) [2525c960] FINISH, HSMClearTaskVDSCommand, log id: 61aabe18 >2012-11-08 10:09:22,051 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-27) [2525c960] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@10a64bc0, log id: 3470074e >2012-11-08 10:09:22,053 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-27) [15b732ca] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:09:22,079 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [15b732ca] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:09:32,133 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [133b9f9d] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:09:32,162 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [133b9f9d] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:09:32,165 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [133b9f9d] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:09:32,177 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [133b9f9d] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 112ac2d9 >2012-11-08 10:09:32,200 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [133b9f9d] spmStart polling started: taskId = 6aa31760-61f0-4398-b2d6-d849ae40aefa >2012-11-08 10:09:33,240 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-72) [133b9f9d] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:09:33,240 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-72) [133b9f9d] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:09:33,240 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [133b9f9d] spmStart polling ended: taskId = 6aa31760-61f0-4398-b2d6-d849ae40aefa task status = finished >2012-11-08 10:09:33,240 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [133b9f9d] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:09:33,268 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [133b9f9d] spmStart polling ended, spm status: Free >2012-11-08 10:09:33,279 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-72) [133b9f9d] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=6aa31760-61f0-4398-b2d6-d849ae40aefa), log id: 66d59a09 >2012-11-08 10:09:33,297 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-72) [133b9f9d] FINISH, HSMClearTaskVDSCommand, log id: 66d59a09 >2012-11-08 10:09:33,297 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [133b9f9d] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@5173f99d, log id: 112ac2d9 >2012-11-08 10:09:33,298 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-72) [1508dc32] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:09:33,316 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [1508dc32] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:09:43,366 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [45dcd2a5] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:09:43,394 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [45dcd2a5] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:09:43,396 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [45dcd2a5] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:09:43,413 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-1) [45dcd2a5] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 34c3e179 >2012-11-08 10:09:43,442 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-1) [45dcd2a5] spmStart polling started: taskId = 3ca9adf5-f70a-4246-897d-a47c702cfb6d >2012-11-08 10:09:44,485 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-1) [45dcd2a5] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:09:44,485 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-1) [45dcd2a5] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:09:44,485 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-1) [45dcd2a5] spmStart polling ended: taskId = 3ca9adf5-f70a-4246-897d-a47c702cfb6d task status = finished >2012-11-08 10:09:44,485 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-1) [45dcd2a5] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:09:44,513 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-1) [45dcd2a5] spmStart polling ended, spm status: Free >2012-11-08 10:09:44,532 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-1) [45dcd2a5] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=3ca9adf5-f70a-4246-897d-a47c702cfb6d), log id: 32534307 >2012-11-08 10:09:44,549 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-1) [45dcd2a5] FINISH, HSMClearTaskVDSCommand, log id: 32534307 >2012-11-08 10:09:44,549 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-1) [45dcd2a5] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@78559a60, log id: 34c3e179 >2012-11-08 10:09:44,552 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-1) [3c0a5258] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:09:44,576 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [3c0a5258] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:09:54,626 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-92) [8fd572e] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:09:54,653 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-92) [8fd572e] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:09:54,656 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-92) [8fd572e] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:09:54,672 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-92) [8fd572e] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 63c8b1cb >2012-11-08 10:09:54,701 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-92) [8fd572e] spmStart polling started: taskId = 4f03e43f-5e0e-4c95-9370-8f49c36b8dfd >2012-11-08 10:09:55,739 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-92) [8fd572e] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:09:55,739 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-92) [8fd572e] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:09:55,739 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-92) [8fd572e] spmStart polling ended: taskId = 4f03e43f-5e0e-4c95-9370-8f49c36b8dfd task status = finished >2012-11-08 10:09:55,739 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-92) [8fd572e] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:09:55,771 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-92) [8fd572e] spmStart polling ended, spm status: Free >2012-11-08 10:09:55,784 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-92) [8fd572e] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=4f03e43f-5e0e-4c95-9370-8f49c36b8dfd), log id: ceb3f7d >2012-11-08 10:09:55,808 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-92) [8fd572e] FINISH, HSMClearTaskVDSCommand, log id: ceb3f7d >2012-11-08 10:09:55,808 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-92) [8fd572e] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@28952e2b, log id: 63c8b1cb >2012-11-08 10:09:55,809 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-92) [5a0a1ddd] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:09:55,829 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-92) [5a0a1ddd] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:10:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-49) [36a7fa37] Autorecovering hosts is disabled, skipping >2012-11-08 10:10:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-49) [36a7fa37] Autorecovering storage domains is disabled, skipping >2012-11-08 10:10:05,888 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [1508dc32] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:10:05,914 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [1508dc32] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:10:05,917 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [1508dc32] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:10:05,933 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [1508dc32] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 185564de >2012-11-08 10:10:05,954 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [1508dc32] spmStart polling started: taskId = ed652811-a4b1-4655-b6ad-39cc9ce3db55 >2012-11-08 10:10:06,986 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-72) [1508dc32] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:10:06,986 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-72) [1508dc32] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:10:06,986 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [1508dc32] spmStart polling ended: taskId = ed652811-a4b1-4655-b6ad-39cc9ce3db55 task status = finished >2012-11-08 10:10:06,986 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [1508dc32] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:10:07,016 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [1508dc32] spmStart polling ended, spm status: Free >2012-11-08 10:10:07,032 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-72) [1508dc32] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=ed652811-a4b1-4655-b6ad-39cc9ce3db55), log id: 50475b9d >2012-11-08 10:10:07,050 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-72) [1508dc32] FINISH, HSMClearTaskVDSCommand, log id: 50475b9d >2012-11-08 10:10:07,050 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [1508dc32] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7ccfc997, log id: 185564de >2012-11-08 10:10:07,053 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-72) [2dbc1885] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:10:07,078 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [2dbc1885] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:10:17,143 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [3c0a5258] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:10:17,171 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [3c0a5258] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:10:17,174 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [3c0a5258] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:10:17,190 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-1) [3c0a5258] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 10aa4f6c >2012-11-08 10:10:17,212 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-1) [3c0a5258] spmStart polling started: taskId = 362c327e-fee8-4bca-a484-30330cc9f042 >2012-11-08 10:10:18,250 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-1) [3c0a5258] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:10:18,250 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-1) [3c0a5258] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:10:18,250 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-1) [3c0a5258] spmStart polling ended: taskId = 362c327e-fee8-4bca-a484-30330cc9f042 task status = finished >2012-11-08 10:10:18,250 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-1) [3c0a5258] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:10:18,281 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-1) [3c0a5258] spmStart polling ended, spm status: Free >2012-11-08 10:10:18,300 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-1) [3c0a5258] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=362c327e-fee8-4bca-a484-30330cc9f042), log id: 4ade49f8 >2012-11-08 10:10:18,324 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-1) [3c0a5258] FINISH, HSMClearTaskVDSCommand, log id: 4ade49f8 >2012-11-08 10:10:18,324 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-1) [3c0a5258] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1c1d03aa, log id: 10aa4f6c >2012-11-08 10:10:18,326 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-1) [34db9c51] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:10:18,344 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [34db9c51] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:10:28,393 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-37) hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:10:28,417 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-37) SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:10:28,420 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-37) starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:10:28,438 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-37) START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2e7ea551 >2012-11-08 10:10:28,465 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-37) spmStart polling started: taskId = 5fc8063b-36a5-46d4-b98b-a371061b99c3 >2012-11-08 10:10:29,506 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-37) Failed in HSMGetTaskStatusVDS method >2012-11-08 10:10:29,507 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-37) Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:10:29,507 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-37) spmStart polling ended: taskId = 5fc8063b-36a5-46d4-b98b-a371061b99c3 task status = finished >2012-11-08 10:10:29,507 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-37) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:10:29,538 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-37) spmStart polling ended, spm status: Free >2012-11-08 10:10:29,553 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-37) START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=5fc8063b-36a5-46d4-b98b-a371061b99c3), log id: 766a6ec0 >2012-11-08 10:10:29,572 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-37) FINISH, HSMClearTaskVDSCommand, log id: 766a6ec0 >2012-11-08 10:10:29,572 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-37) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@458d3bfc, log id: 2e7ea551 >2012-11-08 10:10:29,574 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-37) [3b5d4110] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:10:29,597 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-37) [3b5d4110] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:10:39,669 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-35) [4c3ca9b3] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:10:39,699 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-35) [4c3ca9b3] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:10:39,714 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-35) [4c3ca9b3] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:10:39,731 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-35) [4c3ca9b3] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 53db8843 >2012-11-08 10:10:39,756 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-35) [4c3ca9b3] spmStart polling started: taskId = 6a3667f0-4e0f-46d2-823c-ef1667da76d7 >2012-11-08 10:10:40,789 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-35) [4c3ca9b3] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:10:40,789 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-35) [4c3ca9b3] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:10:40,789 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-35) [4c3ca9b3] spmStart polling ended: taskId = 6a3667f0-4e0f-46d2-823c-ef1667da76d7 task status = finished >2012-11-08 10:10:40,789 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-35) [4c3ca9b3] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:10:40,821 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-35) [4c3ca9b3] spmStart polling ended, spm status: Free >2012-11-08 10:10:40,836 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-35) [4c3ca9b3] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=6a3667f0-4e0f-46d2-823c-ef1667da76d7), log id: 67c1b1af >2012-11-08 10:10:40,858 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-35) [4c3ca9b3] FINISH, HSMClearTaskVDSCommand, log id: 67c1b1af >2012-11-08 10:10:40,858 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-35) [4c3ca9b3] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@5d788d60, log id: 53db8843 >2012-11-08 10:10:40,860 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-35) [1589c991] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:10:40,882 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-35) [1589c991] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:10:50,950 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-6) [b170781] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:10:50,975 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-6) [b170781] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:10:50,978 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-6) [b170781] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:10:50,992 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-6) [b170781] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5e44df6e >2012-11-08 10:10:51,026 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-6) [b170781] spmStart polling started: taskId = f38410db-7d45-439e-8c57-a1db00c7a83a >2012-11-08 10:10:52,063 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-6) [b170781] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:10:52,063 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-6) [b170781] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:10:52,063 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-6) [b170781] spmStart polling ended: taskId = f38410db-7d45-439e-8c57-a1db00c7a83a task status = finished >2012-11-08 10:10:52,063 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-6) [b170781] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:10:52,092 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-6) [b170781] spmStart polling ended, spm status: Free >2012-11-08 10:10:52,104 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-6) [b170781] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=f38410db-7d45-439e-8c57-a1db00c7a83a), log id: 481b9159 >2012-11-08 10:10:52,122 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-6) [b170781] FINISH, HSMClearTaskVDSCommand, log id: 481b9159 >2012-11-08 10:10:52,122 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-6) [b170781] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@9e38f0b, log id: 5e44df6e >2012-11-08 10:10:52,124 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-6) [3ac4203f] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:10:52,151 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-6) [3ac4203f] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:11:02,214 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [2a9ef1af] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:11:02,242 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [2a9ef1af] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:11:02,245 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [2a9ef1af] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:11:02,261 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [2a9ef1af] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 769154bd >2012-11-08 10:11:02,289 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [2a9ef1af] spmStart polling started: taskId = a396c9c2-cf70-4eb7-ba71-54072922bb6a >2012-11-08 10:11:03,333 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-48) [2a9ef1af] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:11:03,334 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-48) [2a9ef1af] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:11:03,334 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [2a9ef1af] spmStart polling ended: taskId = a396c9c2-cf70-4eb7-ba71-54072922bb6a task status = finished >2012-11-08 10:11:03,334 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [2a9ef1af] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:11:03,361 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [2a9ef1af] spmStart polling ended, spm status: Free >2012-11-08 10:11:03,378 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-48) [2a9ef1af] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=a396c9c2-cf70-4eb7-ba71-54072922bb6a), log id: 682dc473 >2012-11-08 10:11:03,403 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-48) [2a9ef1af] FINISH, HSMClearTaskVDSCommand, log id: 682dc473 >2012-11-08 10:11:03,403 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [2a9ef1af] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@14bc8f74, log id: 769154bd >2012-11-08 10:11:03,408 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-48) [676e134e] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:11:03,432 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [676e134e] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:11:13,486 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-93) [2d5ff402] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:11:13,511 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-93) [2d5ff402] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:11:13,513 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-93) [2d5ff402] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:11:13,523 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-93) [2d5ff402] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 77083555 >2012-11-08 10:11:13,544 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-93) [2d5ff402] spmStart polling started: taskId = b7352422-a140-4936-b0db-3c03e9f1d2d3 >2012-11-08 10:11:14,578 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-93) [2d5ff402] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:11:14,578 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-93) [2d5ff402] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:11:14,578 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-93) [2d5ff402] spmStart polling ended: taskId = b7352422-a140-4936-b0db-3c03e9f1d2d3 task status = finished >2012-11-08 10:11:14,578 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-93) [2d5ff402] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:11:14,608 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-93) [2d5ff402] spmStart polling ended, spm status: Free >2012-11-08 10:11:14,626 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-93) [2d5ff402] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=b7352422-a140-4936-b0db-3c03e9f1d2d3), log id: 27dad19b >2012-11-08 10:11:14,646 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-93) [2d5ff402] FINISH, HSMClearTaskVDSCommand, log id: 27dad19b >2012-11-08 10:11:14,646 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-93) [2d5ff402] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@538bf580, log id: 77083555 >2012-11-08 10:11:14,648 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-93) [9a2d15a] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:11:14,674 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-93) [9a2d15a] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:11:24,729 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [4b7f584b] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:11:24,758 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [4b7f584b] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:11:24,761 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [4b7f584b] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:11:24,778 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-84) [4b7f584b] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7c8eeb44 >2012-11-08 10:11:24,802 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-84) [4b7f584b] spmStart polling started: taskId = 1d99df50-3aee-41bc-adab-60797b022fb1 >2012-11-08 10:11:25,832 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-84) [4b7f584b] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:11:25,832 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-84) [4b7f584b] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:11:25,832 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-84) [4b7f584b] spmStart polling ended: taskId = 1d99df50-3aee-41bc-adab-60797b022fb1 task status = finished >2012-11-08 10:11:25,832 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-84) [4b7f584b] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:11:25,861 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-84) [4b7f584b] spmStart polling ended, spm status: Free >2012-11-08 10:11:25,873 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-84) [4b7f584b] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=1d99df50-3aee-41bc-adab-60797b022fb1), log id: 192bfc5c >2012-11-08 10:11:25,891 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-84) [4b7f584b] FINISH, HSMClearTaskVDSCommand, log id: 192bfc5c >2012-11-08 10:11:25,891 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-84) [4b7f584b] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@16c508ed, log id: 7c8eeb44 >2012-11-08 10:11:25,893 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-84) [61cd0b48] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:11:25,912 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [61cd0b48] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:11:35,972 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [2bf5672a] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:11:35,994 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [2bf5672a] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:11:35,996 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [2bf5672a] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:11:36,010 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-42) [2bf5672a] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 14a4416a >2012-11-08 10:11:36,032 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-42) [2bf5672a] spmStart polling started: taskId = a075b986-38c1-46c2-838f-7b3496fe05da >2012-11-08 10:11:37,061 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-42) [2bf5672a] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:11:37,061 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-42) [2bf5672a] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:11:37,061 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-42) [2bf5672a] spmStart polling ended: taskId = a075b986-38c1-46c2-838f-7b3496fe05da task status = finished >2012-11-08 10:11:37,061 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-42) [2bf5672a] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:11:37,089 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-42) [2bf5672a] spmStart polling ended, spm status: Free >2012-11-08 10:11:37,101 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-42) [2bf5672a] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=a075b986-38c1-46c2-838f-7b3496fe05da), log id: 50a32824 >2012-11-08 10:11:37,125 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-42) [2bf5672a] FINISH, HSMClearTaskVDSCommand, log id: 50a32824 >2012-11-08 10:11:37,125 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-42) [2bf5672a] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7a07cd8, log id: 14a4416a >2012-11-08 10:11:37,126 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-42) [39644cd1] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:11:37,148 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-42) [39644cd1] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:11:47,191 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [2dbc1885] starting ProcessDomainRecovery for domain e0cce407-ebea-4322-b47e-7aceb73a7a6f:DISREP >2012-11-08 10:11:47,204 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-76) [2879b44b] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:11:47,229 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-76) [2879b44b] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:11:47,231 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-76) [2879b44b] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:11:47,235 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [2dbc1885] domain e0cce407-ebea-4322-b47e-7aceb73a7a6f:DISREP was reported by all hosts in status UP as problematic. Moving the Domain to NonOperational. >2012-11-08 10:11:47,241 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-76) [2879b44b] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 52001411 >2012-11-08 10:11:47,248 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (QuartzScheduler_Worker-72) [21af292c] Lock Acquired to object EngineLock [exclusiveLocks= key: e0cce407-ebea-4322-b47e-7aceb73a7a6f value: STORAGE >, sharedLocks= key: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 value: POOL >] >2012-11-08 10:11:47,248 WARN [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (QuartzScheduler_Worker-72) [21af292c] CanDoAction of action DeactivateStorageDomain failed. Reasons:VAR__TYPE__STORAGE__DOMAIN,VAR__ACTION__DEACTIVATE,ACTION_TYPE_FAILED_STORAGE_DOMAIN_STATUS_ILLEGAL >2012-11-08 10:11:47,249 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (QuartzScheduler_Worker-72) [21af292c] Lock freed to object EngineLock [exclusiveLocks= key: e0cce407-ebea-4322-b47e-7aceb73a7a6f value: STORAGE >, sharedLocks= key: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 value: POOL >] >2012-11-08 10:11:47,262 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-76) [2879b44b] spmStart polling started: taskId = f6fe18c3-db17-4a12-a640-5816e316ea48 >2012-11-08 10:11:48,296 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-76) [2879b44b] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:11:48,296 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-76) [2879b44b] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:11:48,296 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-76) [2879b44b] spmStart polling ended: taskId = f6fe18c3-db17-4a12-a640-5816e316ea48 task status = finished >2012-11-08 10:11:48,296 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-76) [2879b44b] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:11:48,326 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-76) [2879b44b] spmStart polling ended, spm status: Free >2012-11-08 10:11:48,340 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-76) [2879b44b] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=f6fe18c3-db17-4a12-a640-5816e316ea48), log id: 6f41e546 >2012-11-08 10:11:48,358 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-76) [2879b44b] FINISH, HSMClearTaskVDSCommand, log id: 6f41e546 >2012-11-08 10:11:48,358 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-76) [2879b44b] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@73f08f4a, log id: 52001411 >2012-11-08 10:11:48,359 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-76) [364f7ebd] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:11:48,375 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-76) [364f7ebd] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:11:56,639 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [68eed012] domain e0cce407-ebea-4322-b47e-7aceb73a7a6f:DISREP in problem. vds: rhs-gp-srv11.lab.eng.blr.redhat.com >2012-11-08 10:11:58,425 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-91) [7185235c] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:11:58,454 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-91) [7185235c] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:11:58,456 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-91) [7185235c] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:11:58,470 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-91) [7185235c] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: d54c816 >2012-11-08 10:11:58,493 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-91) [7185235c] spmStart polling started: taskId = 4e854060-06d3-4a0e-9152-4ab0020c1286 >2012-11-08 10:11:59,526 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-91) [7185235c] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:11:59,527 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-91) [7185235c] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:11:59,527 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-91) [7185235c] spmStart polling ended: taskId = 4e854060-06d3-4a0e-9152-4ab0020c1286 task status = finished >2012-11-08 10:11:59,527 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-91) [7185235c] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:11:59,557 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-91) [7185235c] spmStart polling ended, spm status: Free >2012-11-08 10:11:59,575 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-91) [7185235c] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=4e854060-06d3-4a0e-9152-4ab0020c1286), log id: 56026780 >2012-11-08 10:11:59,592 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-91) [7185235c] FINISH, HSMClearTaskVDSCommand, log id: 56026780 >2012-11-08 10:11:59,592 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-91) [7185235c] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@37d9309d, log id: d54c816 >2012-11-08 10:11:59,594 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-91) [1406b010] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:11:59,619 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-91) [1406b010] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:12:08,748 INFO [org.ovirt.engine.core.bll.LoginAdminUserCommand] (ajp-/127.0.0.1:8702-1) Checking if user admin@internal is an admin, result true >2012-11-08 10:12:08,748 INFO [org.ovirt.engine.core.bll.LoginAdminUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginAdminUserCommand internal: false. >2012-11-08 10:12:09,678 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-40) [17449a1f] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:12:09,706 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-40) [17449a1f] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:12:09,709 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-40) [17449a1f] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:12:09,726 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-40) [17449a1f] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 77c4e7f0 >2012-11-08 10:12:09,756 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-40) [17449a1f] spmStart polling started: taskId = 3c040928-b61b-4f54-88dd-ca5dc635abf2 >2012-11-08 10:12:10,801 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-40) [17449a1f] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:12:10,801 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-40) [17449a1f] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:12:10,801 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-40) [17449a1f] spmStart polling ended: taskId = 3c040928-b61b-4f54-88dd-ca5dc635abf2 task status = finished >2012-11-08 10:12:10,801 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-40) [17449a1f] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:12:10,829 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-40) [17449a1f] spmStart polling ended, spm status: Free >2012-11-08 10:12:10,841 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-40) [17449a1f] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=3c040928-b61b-4f54-88dd-ca5dc635abf2), log id: 2f808ab8 >2012-11-08 10:12:10,864 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-40) [17449a1f] FINISH, HSMClearTaskVDSCommand, log id: 2f808ab8 >2012-11-08 10:12:10,864 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-40) [17449a1f] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@20134e65, log id: 77c4e7f0 >2012-11-08 10:12:10,866 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-40) [537fd956] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:12:10,890 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-40) [537fd956] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:12:20,950 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) [2d0064b2] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:12:20,977 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) [2d0064b2] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:12:20,980 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) [2d0064b2] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:12:20,991 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-13) [2d0064b2] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 75fb1169 >2012-11-08 10:12:21,018 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-13) [2d0064b2] spmStart polling started: taskId = 53b38a37-3ad2-4762-beaf-781a6b977dee >2012-11-08 10:12:22,056 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-13) [2d0064b2] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:12:22,058 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-13) [2d0064b2] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:12:22,058 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-13) [2d0064b2] spmStart polling ended: taskId = 53b38a37-3ad2-4762-beaf-781a6b977dee task status = finished >2012-11-08 10:12:22,058 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-13) [2d0064b2] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:12:22,088 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-13) [2d0064b2] spmStart polling ended, spm status: Free >2012-11-08 10:12:22,100 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-13) [2d0064b2] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=53b38a37-3ad2-4762-beaf-781a6b977dee), log id: 4245885 >2012-11-08 10:12:22,119 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-13) [2d0064b2] FINISH, HSMClearTaskVDSCommand, log id: 4245885 >2012-11-08 10:12:22,119 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-13) [2d0064b2] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@74a3b913, log id: 75fb1169 >2012-11-08 10:12:22,120 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-13) [28cbca38] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:12:22,140 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) [28cbca38] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:12:32,187 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [61cd0b48] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:12:32,216 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [61cd0b48] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:12:32,219 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [61cd0b48] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:12:32,236 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-84) [61cd0b48] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2ed85696 >2012-11-08 10:12:32,258 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-84) [61cd0b48] spmStart polling started: taskId = 18312c96-ca21-4a65-be4a-38f15c657a05 >2012-11-08 10:12:33,294 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-84) [61cd0b48] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:12:33,294 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-84) [61cd0b48] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:12:33,294 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-84) [61cd0b48] spmStart polling ended: taskId = 18312c96-ca21-4a65-be4a-38f15c657a05 task status = finished >2012-11-08 10:12:33,294 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-84) [61cd0b48] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:12:33,325 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-84) [61cd0b48] spmStart polling ended, spm status: Free >2012-11-08 10:12:33,342 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-84) [61cd0b48] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=18312c96-ca21-4a65-be4a-38f15c657a05), log id: 44e859d >2012-11-08 10:12:33,362 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-84) [61cd0b48] FINISH, HSMClearTaskVDSCommand, log id: 44e859d >2012-11-08 10:12:33,362 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-84) [61cd0b48] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@41a97803, log id: 2ed85696 >2012-11-08 10:12:33,364 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-84) [5d062ea2] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:12:33,387 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [5d062ea2] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:12:43,434 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-45) [7f87fd9a] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:12:43,463 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-45) [7f87fd9a] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:12:43,467 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-45) [7f87fd9a] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:12:43,479 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-45) [7f87fd9a] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 225f0b8b >2012-11-08 10:12:43,502 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-45) [7f87fd9a] spmStart polling started: taskId = be13167d-0ee1-40f7-94c4-2512eca6dcfa >2012-11-08 10:12:44,534 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-45) [7f87fd9a] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:12:44,534 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-45) [7f87fd9a] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:12:44,534 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-45) [7f87fd9a] spmStart polling ended: taskId = be13167d-0ee1-40f7-94c4-2512eca6dcfa task status = finished >2012-11-08 10:12:44,534 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-45) [7f87fd9a] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:12:44,565 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-45) [7f87fd9a] spmStart polling ended, spm status: Free >2012-11-08 10:12:44,578 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-45) [7f87fd9a] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=be13167d-0ee1-40f7-94c4-2512eca6dcfa), log id: 740c2556 >2012-11-08 10:12:44,596 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-45) [7f87fd9a] FINISH, HSMClearTaskVDSCommand, log id: 740c2556 >2012-11-08 10:12:44,596 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-45) [7f87fd9a] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@49676c9f, log id: 225f0b8b >2012-11-08 10:12:44,598 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-45) [250c605f] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:12:44,619 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-45) [250c605f] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:12:54,678 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-93) [9a2d15a] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:12:54,699 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-93) [9a2d15a] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:12:54,701 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-93) [9a2d15a] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:12:54,716 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-93) [9a2d15a] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3314d93c >2012-11-08 10:12:54,745 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-93) [9a2d15a] spmStart polling started: taskId = 31cf9ddc-95ad-48b7-bd89-652483b453b8 >2012-11-08 10:12:55,785 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-93) [9a2d15a] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:12:55,785 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-93) [9a2d15a] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:12:55,785 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-93) [9a2d15a] spmStart polling ended: taskId = 31cf9ddc-95ad-48b7-bd89-652483b453b8 task status = finished >2012-11-08 10:12:55,785 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-93) [9a2d15a] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:12:55,813 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-93) [9a2d15a] spmStart polling ended, spm status: Free >2012-11-08 10:12:55,832 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-93) [9a2d15a] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=31cf9ddc-95ad-48b7-bd89-652483b453b8), log id: 730a4460 >2012-11-08 10:12:55,852 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-93) [9a2d15a] FINISH, HSMClearTaskVDSCommand, log id: 730a4460 >2012-11-08 10:12:55,852 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-93) [9a2d15a] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@270f35a1, log id: 3314d93c >2012-11-08 10:12:55,854 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-93) [5789f10f] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:12:55,878 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-93) [5789f10f] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:13:05,944 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-18) [37694f38] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:13:05,972 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-18) [37694f38] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:13:05,975 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-18) [37694f38] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:13:05,992 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-18) [37694f38] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 21243d0b >2012-11-08 10:13:06,017 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-18) [37694f38] spmStart polling started: taskId = ffcbd275-f1c4-4d90-94e8-66ae27fc2c93 >2012-11-08 10:13:07,053 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-18) [37694f38] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:13:07,053 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-18) [37694f38] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:13:07,054 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-18) [37694f38] spmStart polling ended: taskId = ffcbd275-f1c4-4d90-94e8-66ae27fc2c93 task status = finished >2012-11-08 10:13:07,054 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-18) [37694f38] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:13:07,078 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-18) [37694f38] spmStart polling ended, spm status: Free >2012-11-08 10:13:07,089 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-18) [37694f38] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=ffcbd275-f1c4-4d90-94e8-66ae27fc2c93), log id: 22818187 >2012-11-08 10:13:07,111 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-18) [37694f38] FINISH, HSMClearTaskVDSCommand, log id: 22818187 >2012-11-08 10:13:07,111 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-18) [37694f38] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@645ed7c3, log id: 21243d0b >2012-11-08 10:13:07,113 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-18) [ea776b0] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:13:07,138 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-18) [ea776b0] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:13:17,204 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-47) [5ce37369] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:13:17,232 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-47) [5ce37369] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:13:17,235 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-47) [5ce37369] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:13:17,252 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-47) [5ce37369] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 594cddef >2012-11-08 10:13:17,281 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-47) [5ce37369] spmStart polling started: taskId = 860a614b-b616-45f4-a74c-3a16bccbb59b >2012-11-08 10:13:18,350 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-47) [5ce37369] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:13:18,350 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-47) [5ce37369] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:13:18,350 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-47) [5ce37369] spmStart polling ended: taskId = 860a614b-b616-45f4-a74c-3a16bccbb59b task status = finished >2012-11-08 10:13:18,350 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-47) [5ce37369] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:13:18,374 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-47) [5ce37369] spmStart polling ended, spm status: Free >2012-11-08 10:13:18,387 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-47) [5ce37369] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=860a614b-b616-45f4-a74c-3a16bccbb59b), log id: 77122e43 >2012-11-08 10:13:18,409 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-47) [5ce37369] FINISH, HSMClearTaskVDSCommand, log id: 77122e43 >2012-11-08 10:13:18,410 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-47) [5ce37369] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4a9b6dc7, log id: 594cddef >2012-11-08 10:13:18,412 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-47) [22444a15] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:13:18,436 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-47) [22444a15] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:13:28,488 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-76) [364f7ebd] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:13:28,515 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-76) [364f7ebd] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:13:28,517 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-76) [364f7ebd] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:13:28,533 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-76) [364f7ebd] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4afbaf3b >2012-11-08 10:13:28,558 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-76) [364f7ebd] spmStart polling started: taskId = 2c7fb0a7-20e1-477e-bf4b-efba289c1adb >2012-11-08 10:13:29,595 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-76) [364f7ebd] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:13:29,595 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-76) [364f7ebd] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:13:29,595 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-76) [364f7ebd] spmStart polling ended: taskId = 2c7fb0a7-20e1-477e-bf4b-efba289c1adb task status = finished >2012-11-08 10:13:29,595 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-76) [364f7ebd] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:13:29,627 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-76) [364f7ebd] spmStart polling ended, spm status: Free >2012-11-08 10:13:29,643 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-76) [364f7ebd] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=2c7fb0a7-20e1-477e-bf4b-efba289c1adb), log id: 45e42507 >2012-11-08 10:13:29,663 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-76) [364f7ebd] FINISH, HSMClearTaskVDSCommand, log id: 45e42507 >2012-11-08 10:13:29,663 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-76) [364f7ebd] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2be627c3, log id: 4afbaf3b >2012-11-08 10:13:29,664 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-76) [133c89b9] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:13:29,688 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-76) [133c89b9] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:13:39,753 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-18) [ea776b0] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:13:39,780 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-18) [ea776b0] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:13:39,781 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-18) [ea776b0] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:13:39,792 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-18) [ea776b0] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 1aa64b3e >2012-11-08 10:13:39,816 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-18) [ea776b0] spmStart polling started: taskId = 8b990df1-6f69-40a7-b405-9a8f74367add >2012-11-08 10:13:40,847 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-18) [ea776b0] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:13:40,848 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-18) [ea776b0] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:13:40,848 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-18) [ea776b0] spmStart polling ended: taskId = 8b990df1-6f69-40a7-b405-9a8f74367add task status = finished >2012-11-08 10:13:40,848 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-18) [ea776b0] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:13:40,875 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-18) [ea776b0] spmStart polling ended, spm status: Free >2012-11-08 10:13:40,892 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-18) [ea776b0] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=8b990df1-6f69-40a7-b405-9a8f74367add), log id: 77605ec6 >2012-11-08 10:13:40,908 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-18) [ea776b0] FINISH, HSMClearTaskVDSCommand, log id: 77605ec6 >2012-11-08 10:13:40,908 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-18) [ea776b0] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@57ee5664, log id: 1aa64b3e >2012-11-08 10:13:40,911 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-18) [6069ec55] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:13:40,938 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-18) [6069ec55] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:13:51,005 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-47) [22444a15] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:13:51,032 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-47) [22444a15] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:13:51,035 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-47) [22444a15] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:13:51,052 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-47) [22444a15] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2adb5a29 >2012-11-08 10:13:51,076 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-47) [22444a15] spmStart polling started: taskId = 1b8250ca-5318-4b98-9e4d-b2b07fa7dd9d >2012-11-08 10:13:52,109 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-47) [22444a15] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:13:52,110 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-47) [22444a15] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:13:52,110 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-47) [22444a15] spmStart polling ended: taskId = 1b8250ca-5318-4b98-9e4d-b2b07fa7dd9d task status = finished >2012-11-08 10:13:52,110 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-47) [22444a15] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:13:52,142 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-47) [22444a15] spmStart polling ended, spm status: Free >2012-11-08 10:13:52,153 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-47) [22444a15] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=1b8250ca-5318-4b98-9e4d-b2b07fa7dd9d), log id: 78f862fd >2012-11-08 10:13:52,172 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-47) [22444a15] FINISH, HSMClearTaskVDSCommand, log id: 78f862fd >2012-11-08 10:13:52,172 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-47) [22444a15] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7b2b0e16, log id: 2adb5a29 >2012-11-08 10:13:52,174 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-47) [47aa0586] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:13:52,197 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-47) [47aa0586] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:14:02,255 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [28842b1b] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:14:02,283 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [28842b1b] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:14:02,286 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [28842b1b] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:14:02,302 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-73) [28842b1b] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5aa508d9 >2012-11-08 10:14:02,326 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-73) [28842b1b] spmStart polling started: taskId = e81d7488-32b7-4dea-88fe-d9546b09b2f7 >2012-11-08 10:14:03,361 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-73) [28842b1b] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:14:03,361 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-73) [28842b1b] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:14:03,361 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-73) [28842b1b] spmStart polling ended: taskId = e81d7488-32b7-4dea-88fe-d9546b09b2f7 task status = finished >2012-11-08 10:14:03,361 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-73) [28842b1b] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:14:03,390 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-73) [28842b1b] spmStart polling ended, spm status: Free >2012-11-08 10:14:03,404 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-73) [28842b1b] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=e81d7488-32b7-4dea-88fe-d9546b09b2f7), log id: 2209f8f8 >2012-11-08 10:14:03,424 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-73) [28842b1b] FINISH, HSMClearTaskVDSCommand, log id: 2209f8f8 >2012-11-08 10:14:03,424 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-73) [28842b1b] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3cb8733d, log id: 5aa508d9 >2012-11-08 10:14:03,426 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-73) [7195c8ef] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:14:03,447 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [7195c8ef] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:14:13,504 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [60ad9f29] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:14:13,531 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [60ad9f29] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:14:13,534 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [60ad9f29] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:14:13,551 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [60ad9f29] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 519436bb >2012-11-08 10:14:13,575 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [60ad9f29] spmStart polling started: taskId = 4090b21c-f137-48e7-8081-bf882909582c >2012-11-08 10:14:14,607 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-16) [60ad9f29] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:14:14,607 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-16) [60ad9f29] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:14:14,607 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [60ad9f29] spmStart polling ended: taskId = 4090b21c-f137-48e7-8081-bf882909582c task status = finished >2012-11-08 10:14:14,607 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [60ad9f29] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:14:14,633 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [60ad9f29] spmStart polling ended, spm status: Free >2012-11-08 10:14:14,645 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-16) [60ad9f29] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=4090b21c-f137-48e7-8081-bf882909582c), log id: 7d18996b >2012-11-08 10:14:14,663 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-16) [60ad9f29] FINISH, HSMClearTaskVDSCommand, log id: 7d18996b >2012-11-08 10:14:14,663 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [60ad9f29] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@5a007ff6, log id: 519436bb >2012-11-08 10:14:14,664 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-16) [56d11da3] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:14:14,684 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [56d11da3] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:14:24,738 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-44) [572b138c] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:14:24,768 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-44) [572b138c] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:14:24,771 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-44) [572b138c] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:14:24,789 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-44) [572b138c] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3e6c6559 >2012-11-08 10:14:24,814 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-44) [572b138c] spmStart polling started: taskId = 129fd284-0a22-4e8e-bb46-e34594cac180 >2012-11-08 10:14:25,848 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-44) [572b138c] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:14:25,848 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-44) [572b138c] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:14:25,848 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-44) [572b138c] spmStart polling ended: taskId = 129fd284-0a22-4e8e-bb46-e34594cac180 task status = finished >2012-11-08 10:14:25,848 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-44) [572b138c] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:14:25,874 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-44) [572b138c] spmStart polling ended, spm status: Free >2012-11-08 10:14:25,890 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-44) [572b138c] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=129fd284-0a22-4e8e-bb46-e34594cac180), log id: 1cbf9444 >2012-11-08 10:14:25,915 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-44) [572b138c] FINISH, HSMClearTaskVDSCommand, log id: 1cbf9444 >2012-11-08 10:14:25,915 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-44) [572b138c] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@659f0b1a, log id: 3e6c6559 >2012-11-08 10:14:25,917 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-44) [2c920c15] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:14:25,936 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-44) [2c920c15] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:14:35,998 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [19bd13e5] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:14:36,028 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [19bd13e5] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:14:36,031 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [19bd13e5] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:14:36,049 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-71) [19bd13e5] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 49e3fcc7 >2012-11-08 10:14:36,079 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-71) [19bd13e5] spmStart polling started: taskId = 40a8a2a4-3e71-4671-a98e-451e5c99d040 >2012-11-08 10:14:37,111 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-71) [19bd13e5] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:14:37,111 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-71) [19bd13e5] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:14:37,111 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-71) [19bd13e5] spmStart polling ended: taskId = 40a8a2a4-3e71-4671-a98e-451e5c99d040 task status = finished >2012-11-08 10:14:37,111 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-71) [19bd13e5] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:14:37,140 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-71) [19bd13e5] spmStart polling ended, spm status: Free >2012-11-08 10:14:37,157 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-71) [19bd13e5] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=40a8a2a4-3e71-4671-a98e-451e5c99d040), log id: 3f50a2b >2012-11-08 10:14:37,174 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-71) [19bd13e5] FINISH, HSMClearTaskVDSCommand, log id: 3f50a2b >2012-11-08 10:14:37,174 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-71) [19bd13e5] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4c6e8f14, log id: 49e3fcc7 >2012-11-08 10:14:37,176 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-71) [6e73913f] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:14:37,199 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [6e73913f] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:14:47,250 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-91) [1406b010] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:14:47,272 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-91) [1406b010] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:14:47,274 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-91) [1406b010] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:14:47,284 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-91) [1406b010] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 77a36ca7 >2012-11-08 10:14:47,305 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-91) [1406b010] spmStart polling started: taskId = 2b8d8dce-3e42-482a-80f0-7427ac4e5749 >2012-11-08 10:14:48,334 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-91) [1406b010] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:14:48,334 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-91) [1406b010] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:14:48,334 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-91) [1406b010] spmStart polling ended: taskId = 2b8d8dce-3e42-482a-80f0-7427ac4e5749 task status = finished >2012-11-08 10:14:48,334 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-91) [1406b010] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:14:48,356 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-91) [1406b010] spmStart polling ended, spm status: Free >2012-11-08 10:14:48,367 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-91) [1406b010] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=2b8d8dce-3e42-482a-80f0-7427ac4e5749), log id: 61365558 >2012-11-08 10:14:48,384 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-91) [1406b010] FINISH, HSMClearTaskVDSCommand, log id: 61365558 >2012-11-08 10:14:48,384 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-91) [1406b010] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@361d3644, log id: 77a36ca7 >2012-11-08 10:14:48,386 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-91) [37d5c61d] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:14:48,401 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-91) [37d5c61d] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:14:58,463 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) [367e311c] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:14:58,490 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) [367e311c] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:14:58,493 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) [367e311c] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:14:58,509 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-55) [367e311c] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 315a8187 >2012-11-08 10:14:58,531 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-55) [367e311c] spmStart polling started: taskId = 4ee90ebd-c99b-48cc-a43a-3e899973701f >2012-11-08 10:14:59,566 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-55) [367e311c] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:14:59,566 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-55) [367e311c] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:14:59,566 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-55) [367e311c] spmStart polling ended: taskId = 4ee90ebd-c99b-48cc-a43a-3e899973701f task status = finished >2012-11-08 10:14:59,566 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-55) [367e311c] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:14:59,596 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-55) [367e311c] spmStart polling ended, spm status: Free >2012-11-08 10:14:59,615 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-55) [367e311c] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=4ee90ebd-c99b-48cc-a43a-3e899973701f), log id: 660c384c >2012-11-08 10:14:59,632 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-55) [367e311c] FINISH, HSMClearTaskVDSCommand, log id: 660c384c >2012-11-08 10:14:59,632 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-55) [367e311c] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@78527a6a, log id: 315a8187 >2012-11-08 10:14:59,635 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-55) [6ee302dc] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:14:59,649 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-55) [6ee302dc] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:15:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-43) [c092d00] Autorecovering hosts is disabled, skipping >2012-11-08 10:15:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-43) [c092d00] Autorecovering storage domains is disabled, skipping >2012-11-08 10:15:09,698 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [4813783b] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:15:09,722 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [4813783b] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:15:09,725 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [4813783b] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:15:09,740 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-77) [4813783b] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5287382f >2012-11-08 10:15:09,761 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-77) [4813783b] spmStart polling started: taskId = da7ce09f-833d-4aa5-953a-8a257d255116 >2012-11-08 10:15:10,792 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-77) [4813783b] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:15:10,792 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-77) [4813783b] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:15:10,792 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-77) [4813783b] spmStart polling ended: taskId = da7ce09f-833d-4aa5-953a-8a257d255116 task status = finished >2012-11-08 10:15:10,792 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-77) [4813783b] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:15:10,820 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-77) [4813783b] spmStart polling ended, spm status: Free >2012-11-08 10:15:10,837 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-77) [4813783b] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=da7ce09f-833d-4aa5-953a-8a257d255116), log id: 55db3bb2 >2012-11-08 10:15:10,854 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-77) [4813783b] FINISH, HSMClearTaskVDSCommand, log id: 55db3bb2 >2012-11-08 10:15:10,854 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-77) [4813783b] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2c2c4f27, log id: 5287382f >2012-11-08 10:15:10,857 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-77) [1100a019] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:15:10,879 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-77) [1100a019] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:15:20,940 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [56d11da3] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:15:20,967 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [56d11da3] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:15:20,970 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [56d11da3] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:15:20,987 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [56d11da3] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 563eee88 >2012-11-08 10:15:21,018 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [56d11da3] spmStart polling started: taskId = 9c61f7d1-75b2-4950-a985-9d0157acdd7a >2012-11-08 10:15:22,057 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-16) [56d11da3] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:15:22,057 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-16) [56d11da3] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:15:22,058 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [56d11da3] spmStart polling ended: taskId = 9c61f7d1-75b2-4950-a985-9d0157acdd7a task status = finished >2012-11-08 10:15:22,058 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [56d11da3] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:15:22,086 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [56d11da3] spmStart polling ended, spm status: Free >2012-11-08 10:15:22,100 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-16) [56d11da3] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=9c61f7d1-75b2-4950-a985-9d0157acdd7a), log id: 13c91c27 >2012-11-08 10:15:22,123 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-16) [56d11da3] FINISH, HSMClearTaskVDSCommand, log id: 13c91c27 >2012-11-08 10:15:22,123 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-16) [56d11da3] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@78574e96, log id: 563eee88 >2012-11-08 10:15:22,125 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-16) [2be89793] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:15:22,142 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-16) [2be89793] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:15:32,199 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-49) [36a7fa37] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:15:32,226 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-49) [36a7fa37] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:15:32,228 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-49) [36a7fa37] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:15:32,246 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-49) [36a7fa37] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 1b1242ea >2012-11-08 10:15:32,275 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-49) [36a7fa37] spmStart polling started: taskId = f89245ce-b4a1-433f-894d-8b656d788da9 >2012-11-08 10:15:33,320 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-49) [36a7fa37] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:15:33,320 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-49) [36a7fa37] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:15:33,320 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-49) [36a7fa37] spmStart polling ended: taskId = f89245ce-b4a1-433f-894d-8b656d788da9 task status = finished >2012-11-08 10:15:33,320 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-49) [36a7fa37] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:15:33,347 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-49) [36a7fa37] spmStart polling ended, spm status: Free >2012-11-08 10:15:33,360 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-49) [36a7fa37] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=f89245ce-b4a1-433f-894d-8b656d788da9), log id: 5486fa52 >2012-11-08 10:15:33,383 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-49) [36a7fa37] FINISH, HSMClearTaskVDSCommand, log id: 5486fa52 >2012-11-08 10:15:33,383 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-49) [36a7fa37] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@699c48d8, log id: 1b1242ea >2012-11-08 10:15:33,385 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-49) [13729313] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:15:33,402 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-49) [13729313] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:15:43,459 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [7195c8ef] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:15:43,486 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [7195c8ef] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:15:43,489 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [7195c8ef] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:15:43,506 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-73) [7195c8ef] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: acdfdc3 >2012-11-08 10:15:43,531 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-73) [7195c8ef] spmStart polling started: taskId = db6a690e-a0ce-4a86-ae06-c933c1858b4d >2012-11-08 10:15:44,559 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-73) [7195c8ef] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:15:44,559 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-73) [7195c8ef] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:15:44,559 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-73) [7195c8ef] spmStart polling ended: taskId = db6a690e-a0ce-4a86-ae06-c933c1858b4d task status = finished >2012-11-08 10:15:44,560 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-73) [7195c8ef] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:15:44,587 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-73) [7195c8ef] spmStart polling ended, spm status: Free >2012-11-08 10:15:44,600 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-73) [7195c8ef] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=db6a690e-a0ce-4a86-ae06-c933c1858b4d), log id: 25ef6cb >2012-11-08 10:15:44,618 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-73) [7195c8ef] FINISH, HSMClearTaskVDSCommand, log id: 25ef6cb >2012-11-08 10:15:44,618 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-73) [7195c8ef] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@289c8aa4, log id: acdfdc3 >2012-11-08 10:15:44,620 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-73) [3d5229d0] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:15:44,637 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-73) [3d5229d0] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:15:54,697 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-28) [52609dc] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:15:54,724 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-28) [52609dc] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:15:54,727 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-28) [52609dc] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:15:54,744 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-28) [52609dc] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 98b859a >2012-11-08 10:15:54,784 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-28) [52609dc] spmStart polling started: taskId = 2e15a3b0-7877-42af-8d7b-edff9fa3a8eb >2012-11-08 10:15:55,831 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-28) [52609dc] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:15:55,831 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-28) [52609dc] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:15:55,831 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-28) [52609dc] spmStart polling ended: taskId = 2e15a3b0-7877-42af-8d7b-edff9fa3a8eb task status = finished >2012-11-08 10:15:55,831 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-28) [52609dc] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:15:55,858 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-28) [52609dc] spmStart polling ended, spm status: Free >2012-11-08 10:15:55,870 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-28) [52609dc] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=2e15a3b0-7877-42af-8d7b-edff9fa3a8eb), log id: dceec69 >2012-11-08 10:15:55,888 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-28) [52609dc] FINISH, HSMClearTaskVDSCommand, log id: dceec69 >2012-11-08 10:15:55,888 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-28) [52609dc] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@c02e894, log id: 98b859a >2012-11-08 10:15:55,890 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-28) [12f352b5] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:15:55,917 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-28) [12f352b5] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:16:05,987 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-47) [47aa0586] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:16:06,014 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-47) [47aa0586] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:16:06,018 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-47) [47aa0586] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:16:06,036 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-47) [47aa0586] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 1d5bc874 >2012-11-08 10:16:06,059 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-47) [47aa0586] spmStart polling started: taskId = 7269eeaa-3f6b-4e45-ac61-d7f1bd32e15c >2012-11-08 10:16:07,100 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-47) [47aa0586] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:16:07,100 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-47) [47aa0586] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:16:07,100 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-47) [47aa0586] spmStart polling ended: taskId = 7269eeaa-3f6b-4e45-ac61-d7f1bd32e15c task status = finished >2012-11-08 10:16:07,100 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-47) [47aa0586] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:16:07,128 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-47) [47aa0586] spmStart polling ended, spm status: Free >2012-11-08 10:16:07,145 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-47) [47aa0586] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=7269eeaa-3f6b-4e45-ac61-d7f1bd32e15c), log id: 41fd956a >2012-11-08 10:16:07,165 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-47) [47aa0586] FINISH, HSMClearTaskVDSCommand, log id: 41fd956a >2012-11-08 10:16:07,165 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-47) [47aa0586] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6f5c83fe, log id: 1d5bc874 >2012-11-08 10:16:07,167 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-47) [22ddd1d0] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:16:07,190 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-47) [22ddd1d0] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:16:17,243 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [5a99d374] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:16:17,273 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [5a99d374] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:16:17,276 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [5a99d374] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:16:17,294 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [5a99d374] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 1ebc6734 >2012-11-08 10:16:17,317 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [5a99d374] spmStart polling started: taskId = 2f415af1-31fe-41ed-a105-3212d051f3af >2012-11-08 10:16:18,351 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-85) [5a99d374] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:16:18,351 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-85) [5a99d374] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:16:18,351 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [5a99d374] spmStart polling ended: taskId = 2f415af1-31fe-41ed-a105-3212d051f3af task status = finished >2012-11-08 10:16:18,351 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [5a99d374] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:16:18,378 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [5a99d374] spmStart polling ended, spm status: Free >2012-11-08 10:16:18,389 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-85) [5a99d374] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=2f415af1-31fe-41ed-a105-3212d051f3af), log id: 388024fa >2012-11-08 10:16:18,408 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-85) [5a99d374] FINISH, HSMClearTaskVDSCommand, log id: 388024fa >2012-11-08 10:16:18,408 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [5a99d374] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@64fbb610, log id: 1ebc6734 >2012-11-08 10:16:18,409 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-85) [496e083b] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:16:18,428 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [496e083b] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:16:28,475 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [2ea13dab] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:16:28,504 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [2ea13dab] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:16:28,506 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [2ea13dab] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:16:28,526 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [2ea13dab] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7d6fb5ff >2012-11-08 10:16:28,550 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [2ea13dab] spmStart polling started: taskId = ba4d0349-d12f-4a64-8340-08a0f9296a03 >2012-11-08 10:16:29,578 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-14) [2ea13dab] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:16:29,578 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-14) [2ea13dab] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:16:29,578 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [2ea13dab] spmStart polling ended: taskId = ba4d0349-d12f-4a64-8340-08a0f9296a03 task status = finished >2012-11-08 10:16:29,578 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [2ea13dab] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:16:29,604 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [2ea13dab] spmStart polling ended, spm status: Free >2012-11-08 10:16:29,617 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-14) [2ea13dab] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=ba4d0349-d12f-4a64-8340-08a0f9296a03), log id: 6ac9869 >2012-11-08 10:16:29,635 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-14) [2ea13dab] FINISH, HSMClearTaskVDSCommand, log id: 6ac9869 >2012-11-08 10:16:29,635 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [2ea13dab] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@733a0234, log id: 7d6fb5ff >2012-11-08 10:16:29,637 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-14) [6566df05] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:16:29,655 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [6566df05] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:16:39,724 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-49) [13729313] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:16:39,752 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-49) [13729313] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:16:39,756 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-49) [13729313] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:16:39,773 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-49) [13729313] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 62eaec84 >2012-11-08 10:16:39,802 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-49) [13729313] spmStart polling started: taskId = df85417b-09e8-4e73-a629-ee740001d53b >2012-11-08 10:16:40,835 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-49) [13729313] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:16:40,835 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-49) [13729313] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:16:40,835 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-49) [13729313] spmStart polling ended: taskId = df85417b-09e8-4e73-a629-ee740001d53b task status = finished >2012-11-08 10:16:40,835 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-49) [13729313] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:16:40,863 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-49) [13729313] spmStart polling ended, spm status: Free >2012-11-08 10:16:40,876 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-49) [13729313] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=df85417b-09e8-4e73-a629-ee740001d53b), log id: 4dccafe3 >2012-11-08 10:16:40,895 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-49) [13729313] FINISH, HSMClearTaskVDSCommand, log id: 4dccafe3 >2012-11-08 10:16:40,895 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-49) [13729313] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@392f87fc, log id: 62eaec84 >2012-11-08 10:16:40,897 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-49) [5f88ceb8] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:16:40,919 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-49) [5f88ceb8] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:16:50,981 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [20c903b7] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:16:51,010 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [20c903b7] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:16:51,013 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [20c903b7] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:16:51,031 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-98) [20c903b7] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5478a089 >2012-11-08 10:16:51,053 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-98) [20c903b7] spmStart polling started: taskId = 69e8fdf8-26bb-438b-811d-fd9015e23560 >2012-11-08 10:16:52,085 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-98) [20c903b7] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:16:52,085 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-98) [20c903b7] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:16:52,086 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-98) [20c903b7] spmStart polling ended: taskId = 69e8fdf8-26bb-438b-811d-fd9015e23560 task status = finished >2012-11-08 10:16:52,086 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-98) [20c903b7] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:16:52,113 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-98) [20c903b7] spmStart polling ended, spm status: Free >2012-11-08 10:16:52,132 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-98) [20c903b7] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=69e8fdf8-26bb-438b-811d-fd9015e23560), log id: bb42ba2 >2012-11-08 10:16:52,149 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-98) [20c903b7] FINISH, HSMClearTaskVDSCommand, log id: bb42ba2 >2012-11-08 10:16:52,149 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-98) [20c903b7] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@619c9eae, log id: 5478a089 >2012-11-08 10:16:52,150 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-98) [7175e392] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:16:52,174 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-98) [7175e392] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:16:56,640 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [68eed012] starting ProcessDomainRecovery for domain e0cce407-ebea-4322-b47e-7aceb73a7a6f:DISREP >2012-11-08 10:16:56,671 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-97) [68eed012] domain e0cce407-ebea-4322-b47e-7aceb73a7a6f:DISREP was reported by all hosts in status UP as problematic. Moving the Domain to NonOperational. >2012-11-08 10:16:56,687 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (QuartzScheduler_Worker-97) [2546c1c9] Lock Acquired to object EngineLock [exclusiveLocks= key: e0cce407-ebea-4322-b47e-7aceb73a7a6f value: STORAGE >, sharedLocks= key: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 value: POOL >] >2012-11-08 10:16:56,687 WARN [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (QuartzScheduler_Worker-97) [2546c1c9] CanDoAction of action DeactivateStorageDomain failed. Reasons:VAR__TYPE__STORAGE__DOMAIN,VAR__ACTION__DEACTIVATE,ACTION_TYPE_FAILED_STORAGE_DOMAIN_STATUS_ILLEGAL >2012-11-08 10:16:56,687 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (QuartzScheduler_Worker-97) [2546c1c9] Lock freed to object EngineLock [exclusiveLocks= key: e0cce407-ebea-4322-b47e-7aceb73a7a6f value: STORAGE >, sharedLocks= key: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 value: POOL >] >2012-11-08 10:17:02,230 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [6a79ef76] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:17:02,260 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [6a79ef76] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:17:02,262 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [6a79ef76] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:17:02,281 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [6a79ef76] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4a516095 >2012-11-08 10:17:02,303 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [6a79ef76] spmStart polling started: taskId = b9dcc5b2-305e-4e1a-98ae-f289f9e12991 >2012-11-08 10:17:03,334 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-23) [6a79ef76] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:17:03,334 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-23) [6a79ef76] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:17:03,334 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [6a79ef76] spmStart polling ended: taskId = b9dcc5b2-305e-4e1a-98ae-f289f9e12991 task status = finished >2012-11-08 10:17:03,334 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [6a79ef76] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:17:03,362 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [6a79ef76] spmStart polling ended, spm status: Free >2012-11-08 10:17:03,376 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-23) [6a79ef76] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=b9dcc5b2-305e-4e1a-98ae-f289f9e12991), log id: 1a13d821 >2012-11-08 10:17:03,394 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-23) [6a79ef76] FINISH, HSMClearTaskVDSCommand, log id: 1a13d821 >2012-11-08 10:17:03,394 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [6a79ef76] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@cd26be7, log id: 4a516095 >2012-11-08 10:17:03,396 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-23) [4aaa1144] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:17:03,415 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [4aaa1144] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:17:05,993 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-26) [4ecd7056] domain e0cce407-ebea-4322-b47e-7aceb73a7a6f:DISREP in problem. vds: rhs-gp-srv11.lab.eng.blr.redhat.com >2012-11-08 10:17:13,466 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-49) [5f88ceb8] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:17:13,493 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-49) [5f88ceb8] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:17:13,496 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-49) [5f88ceb8] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:17:13,513 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-49) [5f88ceb8] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 598bd50c >2012-11-08 10:17:13,542 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-49) [5f88ceb8] spmStart polling started: taskId = a098981b-2eff-46ef-81f6-0d124d9cefbc >2012-11-08 10:17:14,578 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-49) [5f88ceb8] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:17:14,579 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-49) [5f88ceb8] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:17:14,579 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-49) [5f88ceb8] spmStart polling ended: taskId = a098981b-2eff-46ef-81f6-0d124d9cefbc task status = finished >2012-11-08 10:17:14,579 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-49) [5f88ceb8] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:17:14,606 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-49) [5f88ceb8] spmStart polling ended, spm status: Free >2012-11-08 10:17:14,617 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-49) [5f88ceb8] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=a098981b-2eff-46ef-81f6-0d124d9cefbc), log id: 6344bd19 >2012-11-08 10:17:14,639 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-49) [5f88ceb8] FINISH, HSMClearTaskVDSCommand, log id: 6344bd19 >2012-11-08 10:17:14,639 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-49) [5f88ceb8] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6cef9695, log id: 598bd50c >2012-11-08 10:17:14,641 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-49) [65874812] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:17:14,666 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-49) [65874812] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:17:24,725 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [4571e9ce] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:17:24,753 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [4571e9ce] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:17:24,756 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [4571e9ce] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:17:24,772 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-90) [4571e9ce] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 69b8224c >2012-11-08 10:17:24,794 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-90) [4571e9ce] spmStart polling started: taskId = 41f12fec-7c62-411d-bef9-b3597ac03d17 >2012-11-08 10:17:25,836 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-90) [4571e9ce] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:17:25,836 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-90) [4571e9ce] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:17:25,837 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-90) [4571e9ce] spmStart polling ended: taskId = 41f12fec-7c62-411d-bef9-b3597ac03d17 task status = finished >2012-11-08 10:17:25,837 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-90) [4571e9ce] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:17:25,864 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-90) [4571e9ce] spmStart polling ended, spm status: Free >2012-11-08 10:17:25,881 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-90) [4571e9ce] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=41f12fec-7c62-411d-bef9-b3597ac03d17), log id: 135569e >2012-11-08 10:17:25,906 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-90) [4571e9ce] FINISH, HSMClearTaskVDSCommand, log id: 135569e >2012-11-08 10:17:25,906 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-90) [4571e9ce] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@54c99fb0, log id: 69b8224c >2012-11-08 10:17:25,908 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-90) [113074d] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:17:25,923 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [113074d] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:17:35,989 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-24) [61158a6b] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:17:36,019 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-24) [61158a6b] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:17:36,022 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-24) [61158a6b] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:17:36,040 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-24) [61158a6b] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2f60fcdd >2012-11-08 10:17:36,070 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-24) [61158a6b] spmStart polling started: taskId = 129744a7-7505-46cb-b180-f0d8ca4b9361 >2012-11-08 10:17:37,115 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-24) [61158a6b] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:17:37,115 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-24) [61158a6b] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:17:37,115 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-24) [61158a6b] spmStart polling ended: taskId = 129744a7-7505-46cb-b180-f0d8ca4b9361 task status = finished >2012-11-08 10:17:37,115 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-24) [61158a6b] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:17:37,143 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-24) [61158a6b] spmStart polling ended, spm status: Free >2012-11-08 10:17:37,155 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-24) [61158a6b] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=129744a7-7505-46cb-b180-f0d8ca4b9361), log id: 25466685 >2012-11-08 10:17:37,179 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-24) [61158a6b] FINISH, HSMClearTaskVDSCommand, log id: 25466685 >2012-11-08 10:17:37,179 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-24) [61158a6b] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@30f4b8e1, log id: 2f60fcdd >2012-11-08 10:17:37,181 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-24) [1295d4e3] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:17:37,206 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-24) [1295d4e3] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:17:47,264 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) [55c3196f] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:17:47,291 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) [55c3196f] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:17:47,293 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) [55c3196f] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:17:47,310 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-60) [55c3196f] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7566a3f9 >2012-11-08 10:17:47,332 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-60) [55c3196f] spmStart polling started: taskId = 82b2120a-b27d-4adf-a12d-ca8dc283480e >2012-11-08 10:17:48,364 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-60) [55c3196f] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:17:48,364 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-60) [55c3196f] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:17:48,364 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-60) [55c3196f] spmStart polling ended: taskId = 82b2120a-b27d-4adf-a12d-ca8dc283480e task status = finished >2012-11-08 10:17:48,364 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-60) [55c3196f] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:17:48,392 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-60) [55c3196f] spmStart polling ended, spm status: Free >2012-11-08 10:17:48,409 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-60) [55c3196f] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=82b2120a-b27d-4adf-a12d-ca8dc283480e), log id: 7e3f05d8 >2012-11-08 10:17:48,426 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-60) [55c3196f] FINISH, HSMClearTaskVDSCommand, log id: 7e3f05d8 >2012-11-08 10:17:48,426 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-60) [55c3196f] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@5b98e8a8, log id: 7566a3f9 >2012-11-08 10:17:48,429 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-60) [13d29c93] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:17:48,452 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) [13d29c93] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:17:58,505 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) [118f9d41] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:17:58,532 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) [118f9d41] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:17:58,534 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) [118f9d41] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:17:58,551 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-86) [118f9d41] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 641ee43f >2012-11-08 10:17:58,574 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-86) [118f9d41] spmStart polling started: taskId = d0bd2661-fe0b-44bc-9b2b-7dd80090838d >2012-11-08 10:17:59,605 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-86) [118f9d41] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:17:59,605 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-86) [118f9d41] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:17:59,605 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-86) [118f9d41] spmStart polling ended: taskId = d0bd2661-fe0b-44bc-9b2b-7dd80090838d task status = finished >2012-11-08 10:17:59,605 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-86) [118f9d41] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:17:59,633 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-86) [118f9d41] spmStart polling ended, spm status: Free >2012-11-08 10:17:59,644 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-86) [118f9d41] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=d0bd2661-fe0b-44bc-9b2b-7dd80090838d), log id: 631ec8bd >2012-11-08 10:17:59,662 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-86) [118f9d41] FINISH, HSMClearTaskVDSCommand, log id: 631ec8bd >2012-11-08 10:17:59,662 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-86) [118f9d41] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7a0d2321, log id: 641ee43f >2012-11-08 10:17:59,663 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-86) [747d8a20] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:17:59,684 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-86) [747d8a20] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:18:09,749 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [6566df05] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:18:09,774 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [6566df05] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:18:09,777 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [6566df05] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:18:09,792 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [6566df05] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 30c4bb20 >2012-11-08 10:18:09,820 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [6566df05] spmStart polling started: taskId = d8ec31b0-da95-4dd6-90f0-3cebc32955fb >2012-11-08 10:18:10,864 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-14) [6566df05] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:18:10,864 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-14) [6566df05] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:18:10,864 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [6566df05] spmStart polling ended: taskId = d8ec31b0-da95-4dd6-90f0-3cebc32955fb task status = finished >2012-11-08 10:18:10,864 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [6566df05] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:18:10,899 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [6566df05] spmStart polling ended, spm status: Free >2012-11-08 10:18:10,919 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-14) [6566df05] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=d8ec31b0-da95-4dd6-90f0-3cebc32955fb), log id: 2589ee2f >2012-11-08 10:18:10,936 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-14) [6566df05] FINISH, HSMClearTaskVDSCommand, log id: 2589ee2f >2012-11-08 10:18:10,936 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [6566df05] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@27462ea, log id: 30c4bb20 >2012-11-08 10:18:10,937 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-14) [2709f1e8] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:18:10,961 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [2709f1e8] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:18:21,011 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) [13d29c93] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:18:21,038 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) [13d29c93] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:18:21,041 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) [13d29c93] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:18:21,062 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-60) [13d29c93] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 73f809fe >2012-11-08 10:18:21,089 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-60) [13d29c93] spmStart polling started: taskId = ed1741b2-af36-45fc-ac6b-95c7219874d4 >2012-11-08 10:18:22,133 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-60) [13d29c93] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:18:22,133 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-60) [13d29c93] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:18:22,134 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-60) [13d29c93] spmStart polling ended: taskId = ed1741b2-af36-45fc-ac6b-95c7219874d4 task status = finished >2012-11-08 10:18:22,134 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-60) [13d29c93] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:18:22,160 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-60) [13d29c93] spmStart polling ended, spm status: Free >2012-11-08 10:18:22,173 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-60) [13d29c93] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=ed1741b2-af36-45fc-ac6b-95c7219874d4), log id: 61538e37 >2012-11-08 10:18:22,195 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-60) [13d29c93] FINISH, HSMClearTaskVDSCommand, log id: 61538e37 >2012-11-08 10:18:22,195 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-60) [13d29c93] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@363bd42b, log id: 73f809fe >2012-11-08 10:18:22,202 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-60) [11e66ede] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:18:22,218 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) [11e66ede] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:18:32,271 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [113074d] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:18:32,300 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [113074d] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:18:32,303 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [113074d] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:18:32,319 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-90) [113074d] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 481cae64 >2012-11-08 10:18:32,340 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-90) [113074d] spmStart polling started: taskId = 56ffad48-c7c6-4faf-ba0e-e4ce9eba08c2 >2012-11-08 10:18:33,378 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-90) [113074d] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:18:33,378 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-90) [113074d] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:18:33,378 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-90) [113074d] spmStart polling ended: taskId = 56ffad48-c7c6-4faf-ba0e-e4ce9eba08c2 task status = finished >2012-11-08 10:18:33,378 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-90) [113074d] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:18:33,403 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-90) [113074d] spmStart polling ended, spm status: Free >2012-11-08 10:18:33,419 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-90) [113074d] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=56ffad48-c7c6-4faf-ba0e-e4ce9eba08c2), log id: 529f2eb1 >2012-11-08 10:18:33,436 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-90) [113074d] FINISH, HSMClearTaskVDSCommand, log id: 529f2eb1 >2012-11-08 10:18:33,436 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-90) [113074d] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@50b290d7, log id: 481cae64 >2012-11-08 10:18:33,439 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-90) [7b433ca4] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:18:33,462 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-90) [7b433ca4] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:18:43,517 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [4aaa1144] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:18:43,547 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [4aaa1144] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:18:43,549 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [4aaa1144] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:18:43,567 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [4aaa1144] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4cadc3a0 >2012-11-08 10:18:43,597 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [4aaa1144] spmStart polling started: taskId = 924fc436-3b4b-4b23-b573-43dc2de9e9b0 >2012-11-08 10:18:44,635 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-23) [4aaa1144] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:18:44,635 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-23) [4aaa1144] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:18:44,635 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [4aaa1144] spmStart polling ended: taskId = 924fc436-3b4b-4b23-b573-43dc2de9e9b0 task status = finished >2012-11-08 10:18:44,635 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [4aaa1144] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:18:44,662 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [4aaa1144] spmStart polling ended, spm status: Free >2012-11-08 10:18:44,673 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-23) [4aaa1144] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=924fc436-3b4b-4b23-b573-43dc2de9e9b0), log id: 6969eb6e >2012-11-08 10:18:44,691 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-23) [4aaa1144] FINISH, HSMClearTaskVDSCommand, log id: 6969eb6e >2012-11-08 10:18:44,691 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-23) [4aaa1144] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@28dfb8d3, log id: 4cadc3a0 >2012-11-08 10:18:44,692 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-23) [77df399b] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:18:44,710 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) [77df399b] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:18:54,773 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [73f8ef14] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:18:54,797 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [73f8ef14] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:18:54,800 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [73f8ef14] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:18:54,817 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-57) [73f8ef14] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: a8f4bf1 >2012-11-08 10:18:54,848 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-57) [73f8ef14] spmStart polling started: taskId = 9ae898ba-d14c-49bc-a088-6ff2d306ea06 >2012-11-08 10:18:55,881 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-57) [73f8ef14] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:18:55,881 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-57) [73f8ef14] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:18:55,881 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-57) [73f8ef14] spmStart polling ended: taskId = 9ae898ba-d14c-49bc-a088-6ff2d306ea06 task status = finished >2012-11-08 10:18:55,881 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-57) [73f8ef14] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:18:55,910 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-57) [73f8ef14] spmStart polling ended, spm status: Free >2012-11-08 10:18:55,927 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-57) [73f8ef14] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=9ae898ba-d14c-49bc-a088-6ff2d306ea06), log id: 62f058d0 >2012-11-08 10:18:55,944 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-57) [73f8ef14] FINISH, HSMClearTaskVDSCommand, log id: 62f058d0 >2012-11-08 10:18:55,944 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-57) [73f8ef14] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1156f26f, log id: a8f4bf1 >2012-11-08 10:18:55,947 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-57) [63465e4d] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:18:55,971 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-57) [63465e4d] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:19:06,029 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-88) [cb09488] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:19:06,057 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-88) [cb09488] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:19:06,068 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-88) [cb09488] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:19:06,086 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-88) [cb09488] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6cbfa142 >2012-11-08 10:19:06,116 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-88) [cb09488] spmStart polling started: taskId = e88a41de-5822-4210-937c-95be01b7e6f6 >2012-11-08 10:19:07,146 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-88) [cb09488] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:19:07,146 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-88) [cb09488] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:19:07,146 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-88) [cb09488] spmStart polling ended: taskId = e88a41de-5822-4210-937c-95be01b7e6f6 task status = finished >2012-11-08 10:19:07,146 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-88) [cb09488] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:19:07,177 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-88) [cb09488] spmStart polling ended, spm status: Free >2012-11-08 10:19:07,190 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-88) [cb09488] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=e88a41de-5822-4210-937c-95be01b7e6f6), log id: 2490939c >2012-11-08 10:19:07,213 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-88) [cb09488] FINISH, HSMClearTaskVDSCommand, log id: 2490939c >2012-11-08 10:19:07,213 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-88) [cb09488] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3adae0f9, log id: 6cbfa142 >2012-11-08 10:19:07,215 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-88) [77e8199e] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:19:07,232 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-88) [77e8199e] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:19:17,303 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [1de49881] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:19:17,334 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [1de49881] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:19:17,338 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [1de49881] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:19:17,355 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [1de49881] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 705a577c >2012-11-08 10:19:17,377 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [1de49881] spmStart polling started: taskId = 64e74ce2-d953-4481-ac5e-02c8b10a92fd >2012-11-08 10:19:18,414 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-7) [1de49881] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:19:18,414 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-7) [1de49881] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:19:18,414 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [1de49881] spmStart polling ended: taskId = 64e74ce2-d953-4481-ac5e-02c8b10a92fd task status = finished >2012-11-08 10:19:18,414 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [1de49881] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:19:18,445 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [1de49881] spmStart polling ended, spm status: Free >2012-11-08 10:19:18,459 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-7) [1de49881] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=64e74ce2-d953-4481-ac5e-02c8b10a92fd), log id: 3acc1a53 >2012-11-08 10:19:18,482 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-7) [1de49881] FINISH, HSMClearTaskVDSCommand, log id: 3acc1a53 >2012-11-08 10:19:18,482 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-7) [1de49881] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2dd45d4d, log id: 705a577c >2012-11-08 10:19:18,484 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-7) [513981bc] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:19:18,502 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [513981bc] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:19:28,556 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [22f7babe] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:19:28,585 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [22f7babe] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:19:28,587 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [22f7babe] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:19:28,600 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [22f7babe] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 16db75b3 >2012-11-08 10:19:28,623 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [22f7babe] spmStart polling started: taskId = 013e9368-b921-4291-82a8-107ff0b0c80f >2012-11-08 10:19:29,656 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-94) [22f7babe] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:19:29,656 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-94) [22f7babe] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:19:29,656 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [22f7babe] spmStart polling ended: taskId = 013e9368-b921-4291-82a8-107ff0b0c80f task status = finished >2012-11-08 10:19:29,656 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [22f7babe] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:19:29,686 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [22f7babe] spmStart polling ended, spm status: Free >2012-11-08 10:19:29,704 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-94) [22f7babe] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=013e9368-b921-4291-82a8-107ff0b0c80f), log id: f29d368 >2012-11-08 10:19:29,722 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-94) [22f7babe] FINISH, HSMClearTaskVDSCommand, log id: f29d368 >2012-11-08 10:19:29,722 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [22f7babe] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2f6589f, log id: 16db75b3 >2012-11-08 10:19:29,724 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-94) [7b34303d] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:19:29,746 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [7b34303d] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:19:39,805 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-100) [18e40f29] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:19:39,834 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-100) [18e40f29] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:19:39,836 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-100) [18e40f29] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:19:39,854 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-100) [18e40f29] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6a13d20c >2012-11-08 10:19:39,879 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-100) [18e40f29] spmStart polling started: taskId = 970760a2-bca0-4ac7-91d0-3c70522ed080 >2012-11-08 10:19:40,920 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-100) [18e40f29] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:19:40,920 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-100) [18e40f29] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:19:40,920 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-100) [18e40f29] spmStart polling ended: taskId = 970760a2-bca0-4ac7-91d0-3c70522ed080 task status = finished >2012-11-08 10:19:40,920 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-100) [18e40f29] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:19:40,948 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-100) [18e40f29] spmStart polling ended, spm status: Free >2012-11-08 10:19:40,960 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-100) [18e40f29] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=970760a2-bca0-4ac7-91d0-3c70522ed080), log id: 65a87db5 >2012-11-08 10:19:40,982 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-100) [18e40f29] FINISH, HSMClearTaskVDSCommand, log id: 65a87db5 >2012-11-08 10:19:40,982 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-100) [18e40f29] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@18d52a37, log id: 6a13d20c >2012-11-08 10:19:40,983 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-100) [52c68d9d] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:19:41,008 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-100) [52c68d9d] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:19:51,068 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-15) [100e750f] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:19:51,096 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-15) [100e750f] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:19:51,098 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-15) [100e750f] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:19:51,115 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-15) [100e750f] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7bca7e14 >2012-11-08 10:19:51,146 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-15) [100e750f] spmStart polling started: taskId = 035ba971-404c-4323-8e2a-4e7128aff5d6 >2012-11-08 10:19:52,189 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-15) [100e750f] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:19:52,190 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-15) [100e750f] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:19:52,190 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-15) [100e750f] spmStart polling ended: taskId = 035ba971-404c-4323-8e2a-4e7128aff5d6 task status = finished >2012-11-08 10:19:52,190 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-15) [100e750f] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:19:52,217 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-15) [100e750f] spmStart polling ended, spm status: Free >2012-11-08 10:19:52,229 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-15) [100e750f] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=035ba971-404c-4323-8e2a-4e7128aff5d6), log id: 786102d3 >2012-11-08 10:19:52,252 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-15) [100e750f] FINISH, HSMClearTaskVDSCommand, log id: 786102d3 >2012-11-08 10:19:52,252 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-15) [100e750f] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2900a863, log id: 7bca7e14 >2012-11-08 10:19:52,253 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-15) [6a2f109c] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:19:52,278 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-15) [6a2f109c] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:19:55,945 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-36) [14862cb3] DbUserCacheManager::refreshAllUserData() - entered >2012-11-08 10:20:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-60) [11e66ede] Autorecovering hosts is disabled, skipping >2012-11-08 10:20:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-60) [11e66ede] Autorecovering storage domains is disabled, skipping >2012-11-08 10:20:02,338 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-68) [77dbb526] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:20:02,365 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-68) [77dbb526] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:20:02,368 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-68) [77dbb526] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:20:02,384 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-68) [77dbb526] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2f536126 >2012-11-08 10:20:02,413 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-68) [77dbb526] spmStart polling started: taskId = 35f88f20-c489-4390-92c4-e7d4fd82bf91 >2012-11-08 10:20:03,455 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-68) [77dbb526] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:20:03,455 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-68) [77dbb526] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:20:03,455 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-68) [77dbb526] spmStart polling ended: taskId = 35f88f20-c489-4390-92c4-e7d4fd82bf91 task status = finished >2012-11-08 10:20:03,455 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-68) [77dbb526] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:20:03,480 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-68) [77dbb526] spmStart polling ended, spm status: Free >2012-11-08 10:20:03,490 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-68) [77dbb526] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=35f88f20-c489-4390-92c4-e7d4fd82bf91), log id: 22a4451e >2012-11-08 10:20:03,508 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-68) [77dbb526] FINISH, HSMClearTaskVDSCommand, log id: 22a4451e >2012-11-08 10:20:03,508 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-68) [77dbb526] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4eb5c656, log id: 2f536126 >2012-11-08 10:20:03,509 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-68) [7ff06078] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:20:03,530 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-68) [7ff06078] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:20:13,590 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [51e1d0aa] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:20:13,618 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [51e1d0aa] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:20:13,620 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [51e1d0aa] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:20:13,631 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [51e1d0aa] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5617bbab >2012-11-08 10:20:13,653 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [51e1d0aa] spmStart polling started: taskId = e8681b10-92de-49d9-a979-106b678df80b >2012-11-08 10:20:14,690 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-69) [51e1d0aa] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:20:14,690 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-69) [51e1d0aa] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:20:14,690 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [51e1d0aa] spmStart polling ended: taskId = e8681b10-92de-49d9-a979-106b678df80b task status = finished >2012-11-08 10:20:14,690 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [51e1d0aa] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:20:14,721 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [51e1d0aa] spmStart polling ended, spm status: Free >2012-11-08 10:20:14,736 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-69) [51e1d0aa] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=e8681b10-92de-49d9-a979-106b678df80b), log id: 4fae3e2c >2012-11-08 10:20:14,759 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-69) [51e1d0aa] FINISH, HSMClearTaskVDSCommand, log id: 4fae3e2c >2012-11-08 10:20:14,759 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [51e1d0aa] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3873e392, log id: 5617bbab >2012-11-08 10:20:14,761 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-69) [4061a332] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:20:14,780 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [4061a332] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:20:24,855 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-26) [4ecd7056] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:20:24,894 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-26) [4ecd7056] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:20:24,896 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-26) [4ecd7056] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:20:24,913 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-26) [4ecd7056] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 132a4517 >2012-11-08 10:20:24,947 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-26) [4ecd7056] spmStart polling started: taskId = 8a74baf7-401e-46ce-b098-be19b9fd07c3 >2012-11-08 10:20:25,985 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-26) [4ecd7056] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:20:25,986 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-26) [4ecd7056] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:20:25,986 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-26) [4ecd7056] spmStart polling ended: taskId = 8a74baf7-401e-46ce-b098-be19b9fd07c3 task status = finished >2012-11-08 10:20:25,986 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-26) [4ecd7056] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:20:26,010 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-26) [4ecd7056] spmStart polling ended, spm status: Free >2012-11-08 10:20:26,025 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-26) [4ecd7056] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=8a74baf7-401e-46ce-b098-be19b9fd07c3), log id: 5a6af18c >2012-11-08 10:20:26,048 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-26) [4ecd7056] FINISH, HSMClearTaskVDSCommand, log id: 5a6af18c >2012-11-08 10:20:26,048 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-26) [4ecd7056] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@23184cb4, log id: 132a4517 >2012-11-08 10:20:26,050 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-26) [1053cf6c] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:20:26,067 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-26) [1053cf6c] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:20:36,129 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [5a462a3b] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:20:36,155 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [5a462a3b] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:20:36,158 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [5a462a3b] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:20:36,174 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-50) [5a462a3b] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 26208443 >2012-11-08 10:20:36,197 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-50) [5a462a3b] spmStart polling started: taskId = 1293ef8b-bd38-4aa7-a367-23e4a0a1f026 >2012-11-08 10:20:37,238 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-50) [5a462a3b] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:20:37,238 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-50) [5a462a3b] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:20:37,238 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-50) [5a462a3b] spmStart polling ended: taskId = 1293ef8b-bd38-4aa7-a367-23e4a0a1f026 task status = finished >2012-11-08 10:20:37,238 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-50) [5a462a3b] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:20:37,265 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-50) [5a462a3b] spmStart polling ended, spm status: Free >2012-11-08 10:20:37,282 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-50) [5a462a3b] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=1293ef8b-bd38-4aa7-a367-23e4a0a1f026), log id: 7fe978b6 >2012-11-08 10:20:37,302 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-50) [5a462a3b] FINISH, HSMClearTaskVDSCommand, log id: 7fe978b6 >2012-11-08 10:20:37,302 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-50) [5a462a3b] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@38f43655, log id: 26208443 >2012-11-08 10:20:37,303 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-50) [291b23a9] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:20:37,327 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [291b23a9] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:20:47,386 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-89) [5e20ee5f] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:20:47,413 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-89) [5e20ee5f] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:20:47,416 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-89) [5e20ee5f] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:20:47,432 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-89) [5e20ee5f] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6dc84406 >2012-11-08 10:20:47,458 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-89) [5e20ee5f] spmStart polling started: taskId = 415597d0-376d-4a0d-9549-af0b3705b81b >2012-11-08 10:20:48,493 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-89) [5e20ee5f] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:20:48,493 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-89) [5e20ee5f] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:20:48,493 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-89) [5e20ee5f] spmStart polling ended: taskId = 415597d0-376d-4a0d-9549-af0b3705b81b task status = finished >2012-11-08 10:20:48,493 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-89) [5e20ee5f] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:20:48,521 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-89) [5e20ee5f] spmStart polling ended, spm status: Free >2012-11-08 10:20:48,535 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-89) [5e20ee5f] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=415597d0-376d-4a0d-9549-af0b3705b81b), log id: 1631501d >2012-11-08 10:20:48,558 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-89) [5e20ee5f] FINISH, HSMClearTaskVDSCommand, log id: 1631501d >2012-11-08 10:20:48,558 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-89) [5e20ee5f] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7f82ab4c, log id: 6dc84406 >2012-11-08 10:20:48,559 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-89) [5bbc4bfa] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:20:48,577 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-89) [5bbc4bfa] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:20:58,638 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [6878973a] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:20:58,669 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [6878973a] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:20:58,672 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [6878973a] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:20:58,690 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-39) [6878973a] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 58ed7f >2012-11-08 10:20:58,714 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-39) [6878973a] spmStart polling started: taskId = d8f182f1-6e04-452e-a684-8c65b4dde1fa >2012-11-08 10:20:59,745 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-39) [6878973a] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:20:59,745 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-39) [6878973a] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:20:59,745 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-39) [6878973a] spmStart polling ended: taskId = d8f182f1-6e04-452e-a684-8c65b4dde1fa task status = finished >2012-11-08 10:20:59,745 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-39) [6878973a] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:20:59,772 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-39) [6878973a] spmStart polling ended, spm status: Free >2012-11-08 10:20:59,785 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-39) [6878973a] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=d8f182f1-6e04-452e-a684-8c65b4dde1fa), log id: 26493d95 >2012-11-08 10:20:59,802 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-39) [6878973a] FINISH, HSMClearTaskVDSCommand, log id: 26493d95 >2012-11-08 10:20:59,803 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-39) [6878973a] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@126a14d6, log id: 58ed7f >2012-11-08 10:20:59,805 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-39) [727df70b] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:20:59,827 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [727df70b] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:21:09,885 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [64aeb0d5] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:21:09,913 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [64aeb0d5] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:21:09,915 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [64aeb0d5] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:21:09,931 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [64aeb0d5] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3cbfa683 >2012-11-08 10:21:09,954 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [64aeb0d5] spmStart polling started: taskId = 4678cb2f-06f7-4beb-be51-b6deba917e7f >2012-11-08 10:21:10,988 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-66) [64aeb0d5] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:21:10,988 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-66) [64aeb0d5] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:21:10,988 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [64aeb0d5] spmStart polling ended: taskId = 4678cb2f-06f7-4beb-be51-b6deba917e7f task status = finished >2012-11-08 10:21:10,988 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [64aeb0d5] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:21:11,019 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [64aeb0d5] spmStart polling ended, spm status: Free >2012-11-08 10:21:11,038 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-66) [64aeb0d5] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=4678cb2f-06f7-4beb-be51-b6deba917e7f), log id: 4f2282ae >2012-11-08 10:21:11,064 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-66) [64aeb0d5] FINISH, HSMClearTaskVDSCommand, log id: 4f2282ae >2012-11-08 10:21:11,064 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [64aeb0d5] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@62b19618, log id: 3cbfa683 >2012-11-08 10:21:11,066 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-66) [78f53446] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:21:11,086 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [78f53446] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:21:21,152 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [3db31c8a] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:21:21,175 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [3db31c8a] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:21:21,178 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [3db31c8a] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:21:21,197 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [3db31c8a] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 49fa38ca >2012-11-08 10:21:21,228 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [3db31c8a] spmStart polling started: taskId = f18ac227-6667-4c90-b48b-8a66c3aa68e2 >2012-11-08 10:21:22,262 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-5) [3db31c8a] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:21:22,262 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-5) [3db31c8a] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:21:22,262 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [3db31c8a] spmStart polling ended: taskId = f18ac227-6667-4c90-b48b-8a66c3aa68e2 task status = finished >2012-11-08 10:21:22,262 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [3db31c8a] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:21:22,290 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [3db31c8a] spmStart polling ended, spm status: Free >2012-11-08 10:21:22,304 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-5) [3db31c8a] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=f18ac227-6667-4c90-b48b-8a66c3aa68e2), log id: 4a4f8e80 >2012-11-08 10:21:22,324 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-5) [3db31c8a] FINISH, HSMClearTaskVDSCommand, log id: 4a4f8e80 >2012-11-08 10:21:22,325 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [3db31c8a] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2d149b9d, log id: 49fa38ca >2012-11-08 10:21:22,327 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-5) [23f4778a] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:21:22,355 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [23f4778a] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:21:32,417 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) [3f4dc79d] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:21:32,447 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) [3f4dc79d] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:21:32,450 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) [3f4dc79d] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:21:32,469 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-41) [3f4dc79d] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2a43868b >2012-11-08 10:21:32,517 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-41) [3f4dc79d] spmStart polling started: taskId = 7ca3d984-a0e6-4ffa-a623-34144dfa54d4 >2012-11-08 10:21:33,558 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-41) [3f4dc79d] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:21:33,558 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-41) [3f4dc79d] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:21:33,558 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-41) [3f4dc79d] spmStart polling ended: taskId = 7ca3d984-a0e6-4ffa-a623-34144dfa54d4 task status = finished >2012-11-08 10:21:33,558 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-41) [3f4dc79d] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:21:33,589 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-41) [3f4dc79d] spmStart polling ended, spm status: Free >2012-11-08 10:21:33,603 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-41) [3f4dc79d] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=7ca3d984-a0e6-4ffa-a623-34144dfa54d4), log id: 7cd2b1c4 >2012-11-08 10:21:33,626 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-41) [3f4dc79d] FINISH, HSMClearTaskVDSCommand, log id: 7cd2b1c4 >2012-11-08 10:21:33,626 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-41) [3f4dc79d] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6c94fae1, log id: 2a43868b >2012-11-08 10:21:33,628 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-41) [28e30178] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:21:33,645 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) [28e30178] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:21:43,699 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-68) [7ff06078] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:21:43,728 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-68) [7ff06078] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:21:43,731 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-68) [7ff06078] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:21:43,748 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-68) [7ff06078] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5176140a >2012-11-08 10:21:43,770 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-68) [7ff06078] spmStart polling started: taskId = af49d485-5d97-4b66-b5be-9ea8cc30a84d >2012-11-08 10:21:44,802 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-68) [7ff06078] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:21:44,802 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-68) [7ff06078] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:21:44,802 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-68) [7ff06078] spmStart polling ended: taskId = af49d485-5d97-4b66-b5be-9ea8cc30a84d task status = finished >2012-11-08 10:21:44,803 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-68) [7ff06078] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:21:44,830 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-68) [7ff06078] spmStart polling ended, spm status: Free >2012-11-08 10:21:44,848 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-68) [7ff06078] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=af49d485-5d97-4b66-b5be-9ea8cc30a84d), log id: 71c1246 >2012-11-08 10:21:44,867 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-68) [7ff06078] FINISH, HSMClearTaskVDSCommand, log id: 71c1246 >2012-11-08 10:21:44,867 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-68) [7ff06078] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6a5e6ea0, log id: 5176140a >2012-11-08 10:21:44,870 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-68) [5db100ee] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:21:44,889 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-68) [5db100ee] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:21:54,940 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [4061a332] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:21:54,968 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [4061a332] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:21:54,971 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [4061a332] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:21:54,988 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [4061a332] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3ba32fe >2012-11-08 10:21:55,018 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [4061a332] spmStart polling started: taskId = e454c111-5971-4745-8c3c-5868bd5ddb95 >2012-11-08 10:21:56,055 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-69) [4061a332] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:21:56,055 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-69) [4061a332] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:21:56,055 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [4061a332] spmStart polling ended: taskId = e454c111-5971-4745-8c3c-5868bd5ddb95 task status = finished >2012-11-08 10:21:56,055 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [4061a332] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:21:56,082 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [4061a332] spmStart polling ended, spm status: Free >2012-11-08 10:21:56,094 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-69) [4061a332] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=e454c111-5971-4745-8c3c-5868bd5ddb95), log id: 369f3a97 >2012-11-08 10:21:56,116 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-69) [4061a332] FINISH, HSMClearTaskVDSCommand, log id: 369f3a97 >2012-11-08 10:21:56,116 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [4061a332] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7e410a91, log id: 3ba32fe >2012-11-08 10:21:56,117 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-69) [601cf658] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:21:56,133 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [601cf658] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:22:05,994 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [15b732ca] starting ProcessDomainRecovery for domain e0cce407-ebea-4322-b47e-7aceb73a7a6f:DISREP >2012-11-08 10:22:06,015 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) [15b732ca] domain e0cce407-ebea-4322-b47e-7aceb73a7a6f:DISREP was reported by all hosts in status UP as problematic. Moving the Domain to NonOperational. >2012-11-08 10:22:06,031 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (QuartzScheduler_Worker-27) [39d97ddf] Lock Acquired to object EngineLock [exclusiveLocks= key: e0cce407-ebea-4322-b47e-7aceb73a7a6f value: STORAGE >, sharedLocks= key: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 value: POOL >] >2012-11-08 10:22:06,031 WARN [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (QuartzScheduler_Worker-27) [39d97ddf] CanDoAction of action DeactivateStorageDomain failed. Reasons:VAR__TYPE__STORAGE__DOMAIN,VAR__ACTION__DEACTIVATE,ACTION_TYPE_FAILED_STORAGE_DOMAIN_STATUS_ILLEGAL >2012-11-08 10:22:06,031 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (QuartzScheduler_Worker-27) [39d97ddf] Lock freed to object EngineLock [exclusiveLocks= key: e0cce407-ebea-4322-b47e-7aceb73a7a6f value: STORAGE >, sharedLocks= key: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 value: POOL >] >2012-11-08 10:22:06,198 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [727df70b] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:22:06,227 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [727df70b] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:22:06,230 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [727df70b] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:22:06,247 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-39) [727df70b] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 63e683fe >2012-11-08 10:22:06,273 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-39) [727df70b] spmStart polling started: taskId = 70f71179-7994-4f89-a4eb-3889de0c1ee4 >2012-11-08 10:22:07,310 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-39) [727df70b] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:22:07,310 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-39) [727df70b] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:22:07,310 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-39) [727df70b] spmStart polling ended: taskId = 70f71179-7994-4f89-a4eb-3889de0c1ee4 task status = finished >2012-11-08 10:22:07,311 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-39) [727df70b] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:22:07,340 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-39) [727df70b] spmStart polling ended, spm status: Free >2012-11-08 10:22:07,353 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-39) [727df70b] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=70f71179-7994-4f89-a4eb-3889de0c1ee4), log id: c5a231d >2012-11-08 10:22:07,375 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-39) [727df70b] FINISH, HSMClearTaskVDSCommand, log id: c5a231d >2012-11-08 10:22:07,375 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-39) [727df70b] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@71b74b40, log id: 63e683fe >2012-11-08 10:22:07,377 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-39) [38ad9113] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:22:07,395 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [38ad9113] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:22:15,587 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-56) [7784c83b] domain e0cce407-ebea-4322-b47e-7aceb73a7a6f:DISREP in problem. vds: rhs-gp-srv11.lab.eng.blr.redhat.com >2012-11-08 10:22:17,451 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-35) [1589c991] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:22:17,473 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-35) [1589c991] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:22:17,475 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-35) [1589c991] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:22:17,494 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-35) [1589c991] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 64b28f55 >2012-11-08 10:22:17,521 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-35) [1589c991] spmStart polling started: taskId = 70fc7f98-7596-482a-9a95-79320366cf55 >2012-11-08 10:22:18,562 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-35) [1589c991] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:22:18,562 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-35) [1589c991] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:22:18,562 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-35) [1589c991] spmStart polling ended: taskId = 70fc7f98-7596-482a-9a95-79320366cf55 task status = finished >2012-11-08 10:22:18,562 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-35) [1589c991] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:22:18,590 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-35) [1589c991] spmStart polling ended, spm status: Free >2012-11-08 10:22:18,607 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-35) [1589c991] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=70fc7f98-7596-482a-9a95-79320366cf55), log id: 251ff442 >2012-11-08 10:22:18,624 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-35) [1589c991] FINISH, HSMClearTaskVDSCommand, log id: 251ff442 >2012-11-08 10:22:18,624 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-35) [1589c991] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@54dd2c18, log id: 64b28f55 >2012-11-08 10:22:18,626 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-35) [47a072f] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:22:18,650 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-35) [47a072f] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:22:28,709 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [601cf658] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:22:28,732 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [601cf658] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:22:28,734 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [601cf658] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:22:28,753 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [601cf658] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6b759584 >2012-11-08 10:22:28,780 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [601cf658] spmStart polling started: taskId = 55b7136c-34a9-4c34-8edb-d15fd4ff58c5 >2012-11-08 10:22:29,823 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-69) [601cf658] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:22:29,823 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-69) [601cf658] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:22:29,823 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [601cf658] spmStart polling ended: taskId = 55b7136c-34a9-4c34-8edb-d15fd4ff58c5 task status = finished >2012-11-08 10:22:29,823 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [601cf658] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:22:29,851 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [601cf658] spmStart polling ended, spm status: Free >2012-11-08 10:22:29,862 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-69) [601cf658] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=55b7136c-34a9-4c34-8edb-d15fd4ff58c5), log id: 4589909 >2012-11-08 10:22:29,886 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-69) [601cf658] FINISH, HSMClearTaskVDSCommand, log id: 4589909 >2012-11-08 10:22:29,886 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-69) [601cf658] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@54a72019, log id: 6b759584 >2012-11-08 10:22:29,888 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-69) [282c74ba] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:22:29,914 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-69) [282c74ba] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:22:39,990 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-37) [3b5d4110] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:22:40,012 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-37) [3b5d4110] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:22:40,015 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-37) [3b5d4110] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:22:40,034 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-37) [3b5d4110] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 59ce9c5 >2012-11-08 10:22:40,066 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-37) [3b5d4110] spmStart polling started: taskId = cd42ff56-520b-4e2d-9d61-09723dc0db76 >2012-11-08 10:22:41,105 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-37) [3b5d4110] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:22:41,105 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-37) [3b5d4110] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:22:41,106 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-37) [3b5d4110] spmStart polling ended: taskId = cd42ff56-520b-4e2d-9d61-09723dc0db76 task status = finished >2012-11-08 10:22:41,106 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-37) [3b5d4110] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:22:41,137 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-37) [3b5d4110] spmStart polling ended, spm status: Free >2012-11-08 10:22:41,150 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-37) [3b5d4110] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=cd42ff56-520b-4e2d-9d61-09723dc0db76), log id: 472c05a1 >2012-11-08 10:22:41,168 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-37) [3b5d4110] FINISH, HSMClearTaskVDSCommand, log id: 472c05a1 >2012-11-08 10:22:41,168 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-37) [3b5d4110] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1b25b188, log id: 59ce9c5 >2012-11-08 10:22:41,170 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-37) [47aa65a4] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:22:41,192 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-37) [47aa65a4] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:22:51,252 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [78f53446] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:22:51,279 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [78f53446] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:22:51,282 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [78f53446] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:22:51,297 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [78f53446] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 1fad59a7 >2012-11-08 10:22:51,321 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [78f53446] spmStart polling started: taskId = 153a9cfc-26b1-4568-b570-ceb8a54994ed >2012-11-08 10:22:52,355 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-66) [78f53446] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:22:52,356 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-66) [78f53446] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:22:52,356 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [78f53446] spmStart polling ended: taskId = 153a9cfc-26b1-4568-b570-ceb8a54994ed task status = finished >2012-11-08 10:22:52,356 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [78f53446] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:22:52,387 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [78f53446] spmStart polling ended, spm status: Free >2012-11-08 10:22:52,405 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-66) [78f53446] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=153a9cfc-26b1-4568-b570-ceb8a54994ed), log id: 60848b20 >2012-11-08 10:22:52,422 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-66) [78f53446] FINISH, HSMClearTaskVDSCommand, log id: 60848b20 >2012-11-08 10:22:52,422 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-66) [78f53446] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1dca1361, log id: 1fad59a7 >2012-11-08 10:22:52,424 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-66) [3dbe548e] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:22:52,444 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-66) [3dbe548e] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:23:02,508 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [34db9c51] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:23:02,537 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [34db9c51] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:23:02,539 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [34db9c51] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:23:02,556 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-1) [34db9c51] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 11fbd5ef >2012-11-08 10:23:02,581 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-1) [34db9c51] spmStart polling started: taskId = 92626c6d-7c12-4004-b65a-e88113b6bf3f >2012-11-08 10:23:03,617 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-1) [34db9c51] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:23:03,617 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-1) [34db9c51] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:23:03,617 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-1) [34db9c51] spmStart polling ended: taskId = 92626c6d-7c12-4004-b65a-e88113b6bf3f task status = finished >2012-11-08 10:23:03,617 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-1) [34db9c51] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:23:03,646 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-1) [34db9c51] spmStart polling ended, spm status: Free >2012-11-08 10:23:03,659 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-1) [34db9c51] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=92626c6d-7c12-4004-b65a-e88113b6bf3f), log id: 5aa2ea34 >2012-11-08 10:23:03,689 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-1) [34db9c51] FINISH, HSMClearTaskVDSCommand, log id: 5aa2ea34 >2012-11-08 10:23:03,689 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-1) [34db9c51] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@26907fb7, log id: 11fbd5ef >2012-11-08 10:23:03,690 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-1) [58b0db28] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:23:03,706 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [58b0db28] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:23:13,767 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [34b2a1e] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:23:13,795 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [34b2a1e] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:23:13,798 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [34b2a1e] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:23:13,821 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-38) [34b2a1e] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 13d4f97c >2012-11-08 10:23:13,848 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-38) [34b2a1e] spmStart polling started: taskId = 3120174a-440e-4683-8537-493357e7a6db >2012-11-08 10:23:14,891 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-38) [34b2a1e] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:23:14,891 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-38) [34b2a1e] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:23:14,891 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-38) [34b2a1e] spmStart polling ended: taskId = 3120174a-440e-4683-8537-493357e7a6db task status = finished >2012-11-08 10:23:14,891 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-38) [34b2a1e] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:23:14,924 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-38) [34b2a1e] spmStart polling ended, spm status: Free >2012-11-08 10:23:14,934 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-38) [34b2a1e] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=3120174a-440e-4683-8537-493357e7a6db), log id: 402cf5ae >2012-11-08 10:23:14,959 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-38) [34b2a1e] FINISH, HSMClearTaskVDSCommand, log id: 402cf5ae >2012-11-08 10:23:14,959 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-38) [34b2a1e] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@47b5936c, log id: 13d4f97c >2012-11-08 10:23:14,961 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-38) [61e1bc3b] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:23:14,991 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [61e1bc3b] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:23:25,042 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [2141779e] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:23:25,072 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [2141779e] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:23:25,075 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [2141779e] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:23:25,093 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [2141779e] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 576584ac >2012-11-08 10:23:25,123 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [2141779e] spmStart polling started: taskId = fdbf6bdb-a36e-40da-9483-7a3f2c7504ce >2012-11-08 10:23:26,168 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-46) [2141779e] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:23:26,168 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-46) [2141779e] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:23:26,168 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [2141779e] spmStart polling ended: taskId = fdbf6bdb-a36e-40da-9483-7a3f2c7504ce task status = finished >2012-11-08 10:23:26,168 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [2141779e] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:23:26,196 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [2141779e] spmStart polling ended, spm status: Free >2012-11-08 10:23:26,214 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-46) [2141779e] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=fdbf6bdb-a36e-40da-9483-7a3f2c7504ce), log id: 75551d59 >2012-11-08 10:23:26,234 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-46) [2141779e] FINISH, HSMClearTaskVDSCommand, log id: 75551d59 >2012-11-08 10:23:26,234 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [2141779e] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@26f9fa3d, log id: 576584ac >2012-11-08 10:23:26,236 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-46) [659bcec8] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:23:26,259 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [659bcec8] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:23:36,318 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-6) [3ac4203f] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:23:36,345 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-6) [3ac4203f] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:23:36,348 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-6) [3ac4203f] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:23:36,365 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-6) [3ac4203f] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 867207 >2012-11-08 10:23:36,395 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-6) [3ac4203f] spmStart polling started: taskId = e289225a-d24f-44f5-b955-cf3ba3ed531d >2012-11-08 10:23:37,427 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-6) [3ac4203f] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:23:37,427 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-6) [3ac4203f] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:23:37,427 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-6) [3ac4203f] spmStart polling ended: taskId = e289225a-d24f-44f5-b955-cf3ba3ed531d task status = finished >2012-11-08 10:23:37,427 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-6) [3ac4203f] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:23:37,456 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-6) [3ac4203f] spmStart polling ended, spm status: Free >2012-11-08 10:23:37,468 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-6) [3ac4203f] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=e289225a-d24f-44f5-b955-cf3ba3ed531d), log id: a44ac9f >2012-11-08 10:23:37,487 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-6) [3ac4203f] FINISH, HSMClearTaskVDSCommand, log id: a44ac9f >2012-11-08 10:23:37,487 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-6) [3ac4203f] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@22a0fffd, log id: 867207 >2012-11-08 10:23:37,489 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-6) [78064ccd] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:23:37,518 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-6) [78064ccd] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:23:47,578 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-37) [47aa65a4] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:23:47,605 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-37) [47aa65a4] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:23:47,607 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-37) [47aa65a4] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:23:47,624 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-37) [47aa65a4] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 1c3cab3 >2012-11-08 10:23:47,648 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-37) [47aa65a4] spmStart polling started: taskId = f962dbd0-2be6-407b-92bf-af0d0e662eb5 >2012-11-08 10:23:48,682 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-37) [47aa65a4] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:23:48,682 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-37) [47aa65a4] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:23:48,682 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-37) [47aa65a4] spmStart polling ended: taskId = f962dbd0-2be6-407b-92bf-af0d0e662eb5 task status = finished >2012-11-08 10:23:48,682 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-37) [47aa65a4] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:23:48,713 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-37) [47aa65a4] spmStart polling ended, spm status: Free >2012-11-08 10:23:48,726 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-37) [47aa65a4] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=f962dbd0-2be6-407b-92bf-af0d0e662eb5), log id: 6953262b >2012-11-08 10:23:48,750 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-37) [47aa65a4] FINISH, HSMClearTaskVDSCommand, log id: 6953262b >2012-11-08 10:23:48,750 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-37) [47aa65a4] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@51ef0712, log id: 1c3cab3 >2012-11-08 10:23:48,751 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-37) [23b958c4] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:23:48,768 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-37) [23b958c4] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:23:58,834 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [e9c49ed] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:23:58,863 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [e9c49ed] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:23:58,865 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [e9c49ed] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:23:58,883 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [e9c49ed] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4a50c9af >2012-11-08 10:23:58,912 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [e9c49ed] spmStart polling started: taskId = d660c6e3-61f2-4ac7-a7a6-32e0ddb1dbc2 >2012-11-08 10:23:59,950 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-83) [e9c49ed] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:23:59,950 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-83) [e9c49ed] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:23:59,950 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [e9c49ed] spmStart polling ended: taskId = d660c6e3-61f2-4ac7-a7a6-32e0ddb1dbc2 task status = finished >2012-11-08 10:23:59,950 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [e9c49ed] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:23:59,982 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [e9c49ed] spmStart polling ended, spm status: Free >2012-11-08 10:23:59,996 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-83) [e9c49ed] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=d660c6e3-61f2-4ac7-a7a6-32e0ddb1dbc2), log id: 9c1fc60 >2012-11-08 10:24:00,014 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-83) [e9c49ed] FINISH, HSMClearTaskVDSCommand, log id: 9c1fc60 >2012-11-08 10:24:00,014 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [e9c49ed] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1ea007a1, log id: 4a50c9af >2012-11-08 10:24:00,017 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-83) [1c54eabc] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:24:00,039 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [1c54eabc] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:24:10,098 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [58b0db28] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:24:10,124 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [58b0db28] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:24:10,128 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [58b0db28] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:24:10,143 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-1) [58b0db28] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3c404427 >2012-11-08 10:24:10,166 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-1) [58b0db28] spmStart polling started: taskId = 6eb2e00a-606f-4871-85c5-597eb3297bd8 >2012-11-08 10:24:11,195 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-1) [58b0db28] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:24:11,195 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-1) [58b0db28] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:24:11,195 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-1) [58b0db28] spmStart polling ended: taskId = 6eb2e00a-606f-4871-85c5-597eb3297bd8 task status = finished >2012-11-08 10:24:11,195 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-1) [58b0db28] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:24:11,223 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-1) [58b0db28] spmStart polling ended, spm status: Free >2012-11-08 10:24:11,236 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-1) [58b0db28] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=6eb2e00a-606f-4871-85c5-597eb3297bd8), log id: 49018798 >2012-11-08 10:24:11,254 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-1) [58b0db28] FINISH, HSMClearTaskVDSCommand, log id: 49018798 >2012-11-08 10:24:11,254 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-1) [58b0db28] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@36d319a, log id: 3c404427 >2012-11-08 10:24:11,256 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-1) [75d37737] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:24:11,275 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-1) [75d37737] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:24:21,335 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [61e1bc3b] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:24:21,362 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [61e1bc3b] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:24:21,365 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [61e1bc3b] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:24:21,382 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-38) [61e1bc3b] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3d8584b3 >2012-11-08 10:24:21,411 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-38) [61e1bc3b] spmStart polling started: taskId = a85bcb05-7bed-4d15-bb90-2192fc57d982 >2012-11-08 10:24:22,449 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-38) [61e1bc3b] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:24:22,449 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-38) [61e1bc3b] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:24:22,449 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-38) [61e1bc3b] spmStart polling ended: taskId = a85bcb05-7bed-4d15-bb90-2192fc57d982 task status = finished >2012-11-08 10:24:22,449 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-38) [61e1bc3b] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:24:22,488 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-38) [61e1bc3b] spmStart polling ended, spm status: Free >2012-11-08 10:24:22,499 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-38) [61e1bc3b] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=a85bcb05-7bed-4d15-bb90-2192fc57d982), log id: 1954ca9a >2012-11-08 10:24:22,524 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-38) [61e1bc3b] FINISH, HSMClearTaskVDSCommand, log id: 1954ca9a >2012-11-08 10:24:22,524 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-38) [61e1bc3b] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@dc15174, log id: 3d8584b3 >2012-11-08 10:24:22,526 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-38) [eca9dba] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:24:22,552 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [eca9dba] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:24:32,629 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [7bebaf42] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:24:32,654 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [7bebaf42] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:24:32,657 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [7bebaf42] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:24:32,673 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-82) [7bebaf42] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6da42961 >2012-11-08 10:24:32,701 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-82) [7bebaf42] spmStart polling started: taskId = a49b1f22-84cd-4390-a67f-66e78fe80e8d >2012-11-08 10:24:33,738 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-82) [7bebaf42] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:24:33,738 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-82) [7bebaf42] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:24:33,738 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-82) [7bebaf42] spmStart polling ended: taskId = a49b1f22-84cd-4390-a67f-66e78fe80e8d task status = finished >2012-11-08 10:24:33,738 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-82) [7bebaf42] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:24:33,766 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-82) [7bebaf42] spmStart polling ended, spm status: Free >2012-11-08 10:24:33,777 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-82) [7bebaf42] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=a49b1f22-84cd-4390-a67f-66e78fe80e8d), log id: 7578d257 >2012-11-08 10:24:33,799 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-82) [7bebaf42] FINISH, HSMClearTaskVDSCommand, log id: 7578d257 >2012-11-08 10:24:33,799 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-82) [7bebaf42] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4f398a01, log id: 6da42961 >2012-11-08 10:24:33,801 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-82) [4a937049] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:24:33,825 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-82) [4a937049] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:24:43,881 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [5d062ea2] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:24:43,911 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [5d062ea2] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:24:43,913 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [5d062ea2] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:24:43,926 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-84) [5d062ea2] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2b0e565f >2012-11-08 10:24:43,951 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-84) [5d062ea2] spmStart polling started: taskId = 8e52cdc5-8419-4a9d-a6e3-9c1ef51a45e3 >2012-11-08 10:24:44,988 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-84) [5d062ea2] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:24:44,988 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-84) [5d062ea2] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:24:44,988 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-84) [5d062ea2] spmStart polling ended: taskId = 8e52cdc5-8419-4a9d-a6e3-9c1ef51a45e3 task status = finished >2012-11-08 10:24:44,988 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-84) [5d062ea2] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:24:45,015 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-84) [5d062ea2] spmStart polling ended, spm status: Free >2012-11-08 10:24:45,031 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-84) [5d062ea2] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=8e52cdc5-8419-4a9d-a6e3-9c1ef51a45e3), log id: 86b72fb >2012-11-08 10:24:45,049 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-84) [5d062ea2] FINISH, HSMClearTaskVDSCommand, log id: 86b72fb >2012-11-08 10:24:45,049 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-84) [5d062ea2] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4649c10e, log id: 2b0e565f >2012-11-08 10:24:45,052 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-84) [1421a22f] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:24:45,078 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [1421a22f] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:24:55,135 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [676e134e] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:24:55,162 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [676e134e] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:24:55,164 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [676e134e] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:24:55,181 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [676e134e] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7ab1e8ce >2012-11-08 10:24:55,203 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [676e134e] spmStart polling started: taskId = 1eb643b9-7e23-4315-83da-07e68ced96c7 >2012-11-08 10:24:55,811 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (pool-4-thread-45) [250c605f] START, IsValidVDSCommand( storagePoolId = f4c98a0b-0578-485f-90f4-743df3d05d7e, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 183aa3cd >2012-11-08 10:24:55,812 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsValidVDSCommand] (pool-4-thread-45) [250c605f] FINISH, IsValidVDSCommand, return: true, log id: 183aa3cd >2012-11-08 10:24:55,812 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-45) [250c605f] START, GetIsoListVDSCommand( storagePoolId = f4c98a0b-0578-485f-90f4-743df3d05d7e, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 65a5314 >2012-11-08 10:24:55,813 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-45) [250c605f] START, HsmGetIsoListVDSCommand(HostName = rhs-arch-srv2.lab.eng.blr.redhat.com, HostId = cf017816-29a2-11e2-949e-52540035ed48, storagePoolId=f4c98a0b-0578-485f-90f4-743df3d05d7e), log id: ec47b41 >2012-11-08 10:24:55,843 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-45) [250c605f] FINISH, HsmGetIsoListVDSCommand, return: [RHEL6.3-20120613.2-Server-x86_64-DVD1.iso], log id: ec47b41 >2012-11-08 10:24:55,843 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-45) [250c605f] FINISH, GetIsoListVDSCommand, return: [RHEL6.3-20120613.2-Server-x86_64-DVD1.iso], log id: 65a5314 >2012-11-08 10:24:55,846 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-45) [250c605f] Finished automatic refresh process for ISO file type with success, for storage domain id 7a268ac7-857a-4c53-966d-247deb509161. >2012-11-08 10:24:56,234 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-48) [676e134e] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:24:56,234 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-48) [676e134e] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:24:56,234 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [676e134e] spmStart polling ended: taskId = 1eb643b9-7e23-4315-83da-07e68ced96c7 task status = finished >2012-11-08 10:24:56,234 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [676e134e] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:24:56,261 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [676e134e] spmStart polling ended, spm status: Free >2012-11-08 10:24:56,279 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-48) [676e134e] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=1eb643b9-7e23-4315-83da-07e68ced96c7), log id: cd8ab99 >2012-11-08 10:24:56,296 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-48) [676e134e] FINISH, HSMClearTaskVDSCommand, log id: cd8ab99 >2012-11-08 10:24:56,296 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [676e134e] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@64e9d065, log id: 7ab1e8ce >2012-11-08 10:24:56,298 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-48) [3270caa5] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:24:56,321 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [3270caa5] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:25:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-78) Autorecovering hosts is disabled, skipping >2012-11-08 10:25:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-78) Autorecovering storage domains is disabled, skipping >2012-11-08 10:25:06,385 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-79) [1e2f6c7b] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:25:06,412 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-79) [1e2f6c7b] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:25:06,414 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-79) [1e2f6c7b] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:25:06,431 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-79) [1e2f6c7b] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: aa522f4 >2012-11-08 10:25:06,461 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-79) [1e2f6c7b] spmStart polling started: taskId = b0385a19-8a52-476d-ab3a-c8520986c32d >2012-11-08 10:25:07,498 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-79) [1e2f6c7b] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:25:07,498 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-79) [1e2f6c7b] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:25:07,498 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-79) [1e2f6c7b] spmStart polling ended: taskId = b0385a19-8a52-476d-ab3a-c8520986c32d task status = finished >2012-11-08 10:25:07,498 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-79) [1e2f6c7b] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:25:07,530 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-79) [1e2f6c7b] spmStart polling ended, spm status: Free >2012-11-08 10:25:07,543 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-79) [1e2f6c7b] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=b0385a19-8a52-476d-ab3a-c8520986c32d), log id: 25238bb0 >2012-11-08 10:25:07,566 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-79) [1e2f6c7b] FINISH, HSMClearTaskVDSCommand, log id: 25238bb0 >2012-11-08 10:25:07,566 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-79) [1e2f6c7b] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@40a7e2dc, log id: aa522f4 >2012-11-08 10:25:07,568 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-79) [4e1efaad] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:25:07,585 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-79) [4e1efaad] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:25:17,651 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [1421a22f] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:25:17,679 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [1421a22f] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:25:17,681 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [1421a22f] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:25:17,699 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-84) [1421a22f] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 761f3669 >2012-11-08 10:25:17,728 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-84) [1421a22f] spmStart polling started: taskId = 91906020-1895-47cd-94d4-5647cb362085 >2012-11-08 10:25:18,773 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-84) [1421a22f] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:25:18,773 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-84) [1421a22f] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:25:18,773 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-84) [1421a22f] spmStart polling ended: taskId = 91906020-1895-47cd-94d4-5647cb362085 task status = finished >2012-11-08 10:25:18,773 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-84) [1421a22f] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:25:18,800 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-84) [1421a22f] spmStart polling ended, spm status: Free >2012-11-08 10:25:18,813 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-84) [1421a22f] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=91906020-1895-47cd-94d4-5647cb362085), log id: f8f7055 >2012-11-08 10:25:18,835 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-84) [1421a22f] FINISH, HSMClearTaskVDSCommand, log id: f8f7055 >2012-11-08 10:25:18,835 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-84) [1421a22f] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1217e46e, log id: 761f3669 >2012-11-08 10:25:18,837 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-84) [5e95fb09] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:25:18,854 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [5e95fb09] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:25:28,908 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [3270caa5] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:25:28,937 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [3270caa5] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:25:28,940 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [3270caa5] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:25:28,958 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [3270caa5] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: d92f65b >2012-11-08 10:25:28,979 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [3270caa5] spmStart polling started: taskId = 08a02a3d-02bd-4128-a785-f4e808997ff3 >2012-11-08 10:25:30,010 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-48) [3270caa5] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:25:30,010 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-48) [3270caa5] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:25:30,010 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [3270caa5] spmStart polling ended: taskId = 08a02a3d-02bd-4128-a785-f4e808997ff3 task status = finished >2012-11-08 10:25:30,011 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [3270caa5] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:25:30,042 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [3270caa5] spmStart polling ended, spm status: Free >2012-11-08 10:25:30,054 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-48) [3270caa5] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=08a02a3d-02bd-4128-a785-f4e808997ff3), log id: 204e417 >2012-11-08 10:25:30,072 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-48) [3270caa5] FINISH, HSMClearTaskVDSCommand, log id: 204e417 >2012-11-08 10:25:30,072 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [3270caa5] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@f1aa30a, log id: d92f65b >2012-11-08 10:25:30,074 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-48) [1dc5ed45] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:25:30,095 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [1dc5ed45] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:25:40,152 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [6e73913f] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:25:40,180 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [6e73913f] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:25:40,182 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [6e73913f] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:25:40,199 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-71) [6e73913f] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 19e60dd7 >2012-11-08 10:25:40,228 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-71) [6e73913f] spmStart polling started: taskId = 2fa47a3c-f428-4610-9b3f-cfddad8ce00d >2012-11-08 10:25:41,268 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-71) [6e73913f] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:25:41,268 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-71) [6e73913f] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:25:41,268 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-71) [6e73913f] spmStart polling ended: taskId = 2fa47a3c-f428-4610-9b3f-cfddad8ce00d task status = finished >2012-11-08 10:25:41,268 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-71) [6e73913f] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:25:41,298 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-71) [6e73913f] spmStart polling ended, spm status: Free >2012-11-08 10:25:41,311 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-71) [6e73913f] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=2fa47a3c-f428-4610-9b3f-cfddad8ce00d), log id: a3b5f5f >2012-11-08 10:25:41,335 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-71) [6e73913f] FINISH, HSMClearTaskVDSCommand, log id: a3b5f5f >2012-11-08 10:25:41,335 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-71) [6e73913f] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3ff25b38, log id: 19e60dd7 >2012-11-08 10:25:41,336 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-71) [4042da54] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:25:41,353 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-71) [4042da54] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:25:51,417 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [3ed7d34e] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:25:51,447 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [3ed7d34e] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:25:51,449 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [3ed7d34e] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:25:51,467 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [3ed7d34e] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 195a250f >2012-11-08 10:25:51,497 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [3ed7d34e] spmStart polling started: taskId = 52b098ea-3fed-4812-8ca0-5ed6c253fe2c >2012-11-08 10:25:52,535 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-8) [3ed7d34e] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:25:52,535 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-8) [3ed7d34e] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:25:52,535 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [3ed7d34e] spmStart polling ended: taskId = 52b098ea-3fed-4812-8ca0-5ed6c253fe2c task status = finished >2012-11-08 10:25:52,535 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [3ed7d34e] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:25:52,567 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [3ed7d34e] spmStart polling ended, spm status: Free >2012-11-08 10:25:52,580 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-8) [3ed7d34e] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=52b098ea-3fed-4812-8ca0-5ed6c253fe2c), log id: 2c89d129 >2012-11-08 10:25:52,604 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-8) [3ed7d34e] FINISH, HSMClearTaskVDSCommand, log id: 2c89d129 >2012-11-08 10:25:52,604 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-8) [3ed7d34e] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@c02f197, log id: 195a250f >2012-11-08 10:25:52,606 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-8) [1542f6aa] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:25:52,623 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) [1542f6aa] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:26:02,682 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-33) [10ef44aa] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:26:02,709 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-33) [10ef44aa] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:26:02,711 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-33) [10ef44aa] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:26:02,727 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-33) [10ef44aa] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2996951b >2012-11-08 10:26:02,750 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-33) [10ef44aa] spmStart polling started: taskId = 0964a3ac-1f49-4010-a4ac-47ad7af67046 >2012-11-08 10:26:03,785 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-33) [10ef44aa] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:26:03,786 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-33) [10ef44aa] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:26:03,786 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-33) [10ef44aa] spmStart polling ended: taskId = 0964a3ac-1f49-4010-a4ac-47ad7af67046 task status = finished >2012-11-08 10:26:03,786 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-33) [10ef44aa] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:26:03,809 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-33) [10ef44aa] spmStart polling ended, spm status: Free >2012-11-08 10:26:03,827 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-33) [10ef44aa] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=0964a3ac-1f49-4010-a4ac-47ad7af67046), log id: 57ae22c6 >2012-11-08 10:26:03,844 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-33) [10ef44aa] FINISH, HSMClearTaskVDSCommand, log id: 57ae22c6 >2012-11-08 10:26:03,844 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-33) [10ef44aa] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@696d2222, log id: 2996951b >2012-11-08 10:26:03,846 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-33) [7bdc1c40] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:26:03,870 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-33) [7bdc1c40] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:26:13,939 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-76) [133c89b9] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:26:13,967 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-76) [133c89b9] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:26:13,970 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-76) [133c89b9] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:26:13,987 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-76) [133c89b9] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6adda73 >2012-11-08 10:26:14,017 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-76) [133c89b9] spmStart polling started: taskId = 98096773-8b78-43ea-9fab-303cd5a76d16 >2012-11-08 10:26:15,050 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-76) [133c89b9] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:26:15,050 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-76) [133c89b9] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:26:15,051 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-76) [133c89b9] spmStart polling ended: taskId = 98096773-8b78-43ea-9fab-303cd5a76d16 task status = finished >2012-11-08 10:26:15,051 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-76) [133c89b9] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:26:15,082 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-76) [133c89b9] spmStart polling ended, spm status: Free >2012-11-08 10:26:15,093 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-76) [133c89b9] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=98096773-8b78-43ea-9fab-303cd5a76d16), log id: 72d3fb15 >2012-11-08 10:26:15,114 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-76) [133c89b9] FINISH, HSMClearTaskVDSCommand, log id: 72d3fb15 >2012-11-08 10:26:15,114 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-76) [133c89b9] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3b095a98, log id: 6adda73 >2012-11-08 10:26:15,116 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-76) [631acf6f] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:26:15,141 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-76) [631acf6f] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:26:25,207 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [5e95fb09] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:26:25,239 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [5e95fb09] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:26:25,241 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [5e95fb09] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:26:25,257 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-84) [5e95fb09] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 20a2a959 >2012-11-08 10:26:25,279 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-84) [5e95fb09] spmStart polling started: taskId = 5e35fbb9-1737-43c4-89ba-35f0f9028883 >2012-11-08 10:26:26,316 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-84) [5e95fb09] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:26:26,316 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-84) [5e95fb09] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:26:26,316 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-84) [5e95fb09] spmStart polling ended: taskId = 5e35fbb9-1737-43c4-89ba-35f0f9028883 task status = finished >2012-11-08 10:26:26,316 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-84) [5e95fb09] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:26:26,345 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-84) [5e95fb09] spmStart polling ended, spm status: Free >2012-11-08 10:26:26,360 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-84) [5e95fb09] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=5e35fbb9-1737-43c4-89ba-35f0f9028883), log id: 78439296 >2012-11-08 10:26:26,384 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-84) [5e95fb09] FINISH, HSMClearTaskVDSCommand, log id: 78439296 >2012-11-08 10:26:26,384 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-84) [5e95fb09] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@505ca0ef, log id: 20a2a959 >2012-11-08 10:26:26,386 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-84) [7149d420] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:26:26,404 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [7149d420] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:26:36,466 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [1dc5ed45] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:26:36,490 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [1dc5ed45] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:26:36,493 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [1dc5ed45] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:26:36,512 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [1dc5ed45] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3b8586ea >2012-11-08 10:26:36,538 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [1dc5ed45] spmStart polling started: taskId = 4aee5148-b822-4936-9fc8-f195f57daa50 >2012-11-08 10:26:37,572 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-48) [1dc5ed45] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:26:37,572 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-48) [1dc5ed45] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:26:37,572 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [1dc5ed45] spmStart polling ended: taskId = 4aee5148-b822-4936-9fc8-f195f57daa50 task status = finished >2012-11-08 10:26:37,572 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [1dc5ed45] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:26:37,599 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [1dc5ed45] spmStart polling ended, spm status: Free >2012-11-08 10:26:37,611 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-48) [1dc5ed45] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=4aee5148-b822-4936-9fc8-f195f57daa50), log id: 44c628bf >2012-11-08 10:26:37,629 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-48) [1dc5ed45] FINISH, HSMClearTaskVDSCommand, log id: 44c628bf >2012-11-08 10:26:37,629 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [1dc5ed45] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2d9d36df, log id: 3b8586ea >2012-11-08 10:26:37,630 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-48) [2c7d81de] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:26:37,649 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [2c7d81de] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:26:47,704 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-79) [4e1efaad] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:26:47,734 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-79) [4e1efaad] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:26:47,737 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-79) [4e1efaad] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:26:47,756 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-79) [4e1efaad] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 63d8bb5a >2012-11-08 10:26:47,780 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-79) [4e1efaad] spmStart polling started: taskId = e42e2591-a331-44c4-bd24-b4d4d622620a >2012-11-08 10:26:48,813 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-79) [4e1efaad] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:26:48,813 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-79) [4e1efaad] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:26:48,813 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-79) [4e1efaad] spmStart polling ended: taskId = e42e2591-a331-44c4-bd24-b4d4d622620a task status = finished >2012-11-08 10:26:48,814 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-79) [4e1efaad] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:26:48,845 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-79) [4e1efaad] spmStart polling ended, spm status: Free >2012-11-08 10:26:48,863 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-79) [4e1efaad] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=e42e2591-a331-44c4-bd24-b4d4d622620a), log id: 432f91a2 >2012-11-08 10:26:48,890 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-79) [4e1efaad] FINISH, HSMClearTaskVDSCommand, log id: 432f91a2 >2012-11-08 10:26:48,890 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-79) [4e1efaad] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@68024523, log id: 63d8bb5a >2012-11-08 10:26:48,892 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-79) [6d09322d] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:26:48,916 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-79) [6d09322d] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:26:58,978 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) [5cd549cb] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:26:59,006 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) [5cd549cb] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:26:59,008 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) [5cd549cb] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:26:59,025 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-51) [5cd549cb] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 36c9f16e >2012-11-08 10:26:59,054 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-51) [5cd549cb] spmStart polling started: taskId = 424233b4-7be3-40d9-8207-1c279312d4fa >2012-11-08 10:27:00,086 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-51) [5cd549cb] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:27:00,087 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-51) [5cd549cb] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:27:00,087 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-51) [5cd549cb] spmStart polling ended: taskId = 424233b4-7be3-40d9-8207-1c279312d4fa task status = finished >2012-11-08 10:27:00,087 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-51) [5cd549cb] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:27:00,116 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-51) [5cd549cb] spmStart polling ended, spm status: Free >2012-11-08 10:27:00,129 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-51) [5cd549cb] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=424233b4-7be3-40d9-8207-1c279312d4fa), log id: 18c174c2 >2012-11-08 10:27:00,151 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-51) [5cd549cb] FINISH, HSMClearTaskVDSCommand, log id: 18c174c2 >2012-11-08 10:27:00,151 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-51) [5cd549cb] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@44d87940, log id: 36c9f16e >2012-11-08 10:27:00,153 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-51) [60007c0f] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:27:00,175 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-51) [60007c0f] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:27:10,233 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-33) [7bdc1c40] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:27:10,262 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-33) [7bdc1c40] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:27:10,265 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-33) [7bdc1c40] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:27:10,282 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-33) [7bdc1c40] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 1829a80f >2012-11-08 10:27:10,305 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-33) [7bdc1c40] spmStart polling started: taskId = 41ad47f7-0579-4fe3-8f6c-9f8d06bdb810 >2012-11-08 10:27:11,336 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-33) [7bdc1c40] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:27:11,337 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-33) [7bdc1c40] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:27:11,337 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-33) [7bdc1c40] spmStart polling ended: taskId = 41ad47f7-0579-4fe3-8f6c-9f8d06bdb810 task status = finished >2012-11-08 10:27:11,337 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-33) [7bdc1c40] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:27:11,367 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-33) [7bdc1c40] spmStart polling ended, spm status: Free >2012-11-08 10:27:11,384 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-33) [7bdc1c40] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=41ad47f7-0579-4fe3-8f6c-9f8d06bdb810), log id: a808604 >2012-11-08 10:27:11,401 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-33) [7bdc1c40] FINISH, HSMClearTaskVDSCommand, log id: a808604 >2012-11-08 10:27:11,401 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-33) [7bdc1c40] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7bde75fe, log id: 1829a80f >2012-11-08 10:27:11,403 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-33) [4a14b63a] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:27:11,425 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-33) [4a14b63a] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:27:15,589 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-70) [c5e7fc0] starting ProcessDomainRecovery for domain e0cce407-ebea-4322-b47e-7aceb73a7a6f:DISREP >2012-11-08 10:27:15,616 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-70) [c5e7fc0] domain e0cce407-ebea-4322-b47e-7aceb73a7a6f:DISREP was reported by all hosts in status UP as problematic. Moving the Domain to NonOperational. >2012-11-08 10:27:15,631 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (QuartzScheduler_Worker-70) [544c5258] Lock Acquired to object EngineLock [exclusiveLocks= key: e0cce407-ebea-4322-b47e-7aceb73a7a6f value: STORAGE >, sharedLocks= key: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 value: POOL >] >2012-11-08 10:27:15,631 WARN [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (QuartzScheduler_Worker-70) [544c5258] CanDoAction of action DeactivateStorageDomain failed. Reasons:VAR__TYPE__STORAGE__DOMAIN,VAR__ACTION__DEACTIVATE,ACTION_TYPE_FAILED_STORAGE_DOMAIN_STATUS_ILLEGAL >2012-11-08 10:27:15,632 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (QuartzScheduler_Worker-70) [544c5258] Lock freed to object EngineLock [exclusiveLocks= key: e0cce407-ebea-4322-b47e-7aceb73a7a6f value: STORAGE >, sharedLocks= key: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 value: POOL >] >2012-11-08 10:27:21,478 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-79) [6d09322d] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:27:21,501 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-79) [6d09322d] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:27:21,504 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-79) [6d09322d] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:27:21,522 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-79) [6d09322d] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7e83e231 >2012-11-08 10:27:21,553 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-79) [6d09322d] spmStart polling started: taskId = c4f63e2e-32a3-47c7-a764-5dc6da39ea34 >2012-11-08 10:27:22,590 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-79) [6d09322d] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:27:22,590 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-79) [6d09322d] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:27:22,590 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-79) [6d09322d] spmStart polling ended: taskId = c4f63e2e-32a3-47c7-a764-5dc6da39ea34 task status = finished >2012-11-08 10:27:22,590 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-79) [6d09322d] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:27:22,617 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-79) [6d09322d] spmStart polling ended, spm status: Free >2012-11-08 10:27:22,629 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-79) [6d09322d] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=c4f63e2e-32a3-47c7-a764-5dc6da39ea34), log id: 7e6d4727 >2012-11-08 10:27:22,651 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-79) [6d09322d] FINISH, HSMClearTaskVDSCommand, log id: 7e6d4727 >2012-11-08 10:27:22,651 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-79) [6d09322d] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@385a9249, log id: 7e83e231 >2012-11-08 10:27:22,653 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-79) [423a1386] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:27:22,677 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-79) [423a1386] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:27:25,048 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-62) [3e48002e] domain e0cce407-ebea-4322-b47e-7aceb73a7a6f:DISREP in problem. vds: rhs-gp-srv11.lab.eng.blr.redhat.com >2012-11-08 10:27:32,746 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-28) [12f352b5] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:27:32,777 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-28) [12f352b5] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:27:32,780 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-28) [12f352b5] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:27:32,795 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-28) [12f352b5] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7e940ca >2012-11-08 10:27:32,835 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-28) [12f352b5] spmStart polling started: taskId = ca301302-4057-448c-b2c9-e72c6f97458b >2012-11-08 10:27:33,870 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-28) [12f352b5] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:27:33,871 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-28) [12f352b5] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:27:33,871 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-28) [12f352b5] spmStart polling ended: taskId = ca301302-4057-448c-b2c9-e72c6f97458b task status = finished >2012-11-08 10:27:33,871 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-28) [12f352b5] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:27:33,911 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-28) [12f352b5] spmStart polling ended, spm status: Free >2012-11-08 10:27:33,925 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-28) [12f352b5] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=ca301302-4057-448c-b2c9-e72c6f97458b), log id: 20915b12 >2012-11-08 10:27:33,949 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-28) [12f352b5] FINISH, HSMClearTaskVDSCommand, log id: 20915b12 >2012-11-08 10:27:33,949 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-28) [12f352b5] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@26f20b72, log id: 7e940ca >2012-11-08 10:27:33,951 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-28) [5ccd7d57] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:27:33,976 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-28) [5ccd7d57] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:27:44,037 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [2c7d81de] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:27:44,064 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [2c7d81de] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:27:44,067 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [2c7d81de] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:27:44,083 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [2c7d81de] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 55b0e54c >2012-11-08 10:27:44,105 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [2c7d81de] spmStart polling started: taskId = 14cb043d-0c77-4b0c-b92a-52c28de6c9a1 >2012-11-08 10:27:45,136 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-48) [2c7d81de] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:27:45,136 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-48) [2c7d81de] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:27:45,136 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [2c7d81de] spmStart polling ended: taskId = 14cb043d-0c77-4b0c-b92a-52c28de6c9a1 task status = finished >2012-11-08 10:27:45,136 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [2c7d81de] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:27:45,164 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [2c7d81de] spmStart polling ended, spm status: Free >2012-11-08 10:27:45,183 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-48) [2c7d81de] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=14cb043d-0c77-4b0c-b92a-52c28de6c9a1), log id: 309066db >2012-11-08 10:27:45,200 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-48) [2c7d81de] FINISH, HSMClearTaskVDSCommand, log id: 309066db >2012-11-08 10:27:45,200 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-48) [2c7d81de] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@5ec215cd, log id: 55b0e54c >2012-11-08 10:27:45,202 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-48) [18257655] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:27:45,226 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-48) [18257655] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:27:55,286 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-76) [631acf6f] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:27:55,313 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-76) [631acf6f] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:27:55,315 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-76) [631acf6f] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:27:55,332 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-76) [631acf6f] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 1d3bb651 >2012-11-08 10:27:55,354 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-76) [631acf6f] spmStart polling started: taskId = 08b3bd4c-2cdf-406f-a79f-ab5ae345dff2 >2012-11-08 10:27:56,390 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-76) [631acf6f] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:27:56,390 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-76) [631acf6f] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:27:56,390 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-76) [631acf6f] spmStart polling ended: taskId = 08b3bd4c-2cdf-406f-a79f-ab5ae345dff2 task status = finished >2012-11-08 10:27:56,390 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-76) [631acf6f] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:27:56,421 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-76) [631acf6f] spmStart polling ended, spm status: Free >2012-11-08 10:27:56,435 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-76) [631acf6f] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=08b3bd4c-2cdf-406f-a79f-ab5ae345dff2), log id: 530646c7 >2012-11-08 10:27:56,453 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-76) [631acf6f] FINISH, HSMClearTaskVDSCommand, log id: 530646c7 >2012-11-08 10:27:56,453 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-76) [631acf6f] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4110dd66, log id: 1d3bb651 >2012-11-08 10:27:56,455 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-76) [3a3774e9] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:27:56,476 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-76) [3a3774e9] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:28:06,528 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-24) [1295d4e3] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:28:06,555 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-24) [1295d4e3] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:28:06,558 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-24) [1295d4e3] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:28:06,573 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-24) [1295d4e3] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2bddc9d3 >2012-11-08 10:28:06,596 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-24) [1295d4e3] spmStart polling started: taskId = 1931f4d5-b594-47e8-81db-98e002300bbb >2012-11-08 10:28:07,632 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-24) [1295d4e3] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:28:07,632 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-24) [1295d4e3] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:28:07,632 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-24) [1295d4e3] spmStart polling ended: taskId = 1931f4d5-b594-47e8-81db-98e002300bbb task status = finished >2012-11-08 10:28:07,632 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-24) [1295d4e3] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:28:07,656 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-24) [1295d4e3] spmStart polling ended, spm status: Free >2012-11-08 10:28:07,672 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-24) [1295d4e3] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=1931f4d5-b594-47e8-81db-98e002300bbb), log id: 1fc1eae2 >2012-11-08 10:28:07,691 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-24) [1295d4e3] FINISH, HSMClearTaskVDSCommand, log id: 1fc1eae2 >2012-11-08 10:28:07,691 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-24) [1295d4e3] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7423abb7, log id: 2bddc9d3 >2012-11-08 10:28:07,694 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-24) [51deb6f2] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:28:07,718 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-24) [51deb6f2] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:28:17,782 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-52) [3dfbeff4] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:28:17,811 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-52) [3dfbeff4] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:28:17,814 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-52) [3dfbeff4] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:28:17,831 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-52) [3dfbeff4] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 8dfc860 >2012-11-08 10:28:17,861 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-52) [3dfbeff4] spmStart polling started: taskId = b4125dc1-6764-4285-90df-070d343dd70a >2012-11-08 10:28:18,898 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-52) [3dfbeff4] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:28:18,898 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-52) [3dfbeff4] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:28:18,898 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-52) [3dfbeff4] spmStart polling ended: taskId = b4125dc1-6764-4285-90df-070d343dd70a task status = finished >2012-11-08 10:28:18,898 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-52) [3dfbeff4] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:28:18,930 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-52) [3dfbeff4] spmStart polling ended, spm status: Free >2012-11-08 10:28:18,949 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-52) [3dfbeff4] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=b4125dc1-6764-4285-90df-070d343dd70a), log id: 2875a224 >2012-11-08 10:28:18,975 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-52) [3dfbeff4] FINISH, HSMClearTaskVDSCommand, log id: 2875a224 >2012-11-08 10:28:18,975 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-52) [3dfbeff4] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@42e7fe3c, log id: 8dfc860 >2012-11-08 10:28:18,977 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-52) [ce43775] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:28:19,003 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-52) [ce43775] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:28:29,057 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-79) [423a1386] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:28:29,083 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-79) [423a1386] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:28:29,087 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-79) [423a1386] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:28:29,102 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-79) [423a1386] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 176d706a >2012-11-08 10:28:29,124 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-79) [423a1386] spmStart polling started: taskId = 0d9a8718-f42d-4b09-8a35-e6f43cd6bb67 >2012-11-08 10:28:30,161 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-79) [423a1386] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:28:30,161 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-79) [423a1386] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:28:30,161 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-79) [423a1386] spmStart polling ended: taskId = 0d9a8718-f42d-4b09-8a35-e6f43cd6bb67 task status = finished >2012-11-08 10:28:30,161 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-79) [423a1386] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:28:30,186 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-79) [423a1386] spmStart polling ended, spm status: Free >2012-11-08 10:28:30,202 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-79) [423a1386] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=0d9a8718-f42d-4b09-8a35-e6f43cd6bb67), log id: 300c2f7d >2012-11-08 10:28:30,219 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-79) [423a1386] FINISH, HSMClearTaskVDSCommand, log id: 300c2f7d >2012-11-08 10:28:30,219 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-79) [423a1386] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7001d02f, log id: 176d706a >2012-11-08 10:28:30,222 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-79) [70af71c] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:28:30,243 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-79) [70af71c] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:28:40,312 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [7149d420] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:28:40,341 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [7149d420] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:28:40,344 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [7149d420] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:28:40,361 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-84) [7149d420] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3f1c9d0c >2012-11-08 10:28:40,390 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-84) [7149d420] spmStart polling started: taskId = 8b78b6ca-6772-4c9c-a0a3-8b18f172c4bd >2012-11-08 10:28:41,426 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-84) [7149d420] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:28:41,427 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-84) [7149d420] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:28:41,427 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-84) [7149d420] spmStart polling ended: taskId = 8b78b6ca-6772-4c9c-a0a3-8b18f172c4bd task status = finished >2012-11-08 10:28:41,427 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-84) [7149d420] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:28:41,453 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-84) [7149d420] spmStart polling ended, spm status: Free >2012-11-08 10:28:41,465 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-84) [7149d420] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=8b78b6ca-6772-4c9c-a0a3-8b18f172c4bd), log id: 42bb2be7 >2012-11-08 10:28:41,487 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-84) [7149d420] FINISH, HSMClearTaskVDSCommand, log id: 42bb2be7 >2012-11-08 10:28:41,487 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-84) [7149d420] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@e3f6ceb, log id: 3f1c9d0c >2012-11-08 10:28:41,489 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-84) [6607bf9b] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:28:41,513 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [6607bf9b] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:28:51,575 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-33) [4a14b63a] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:28:51,602 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-33) [4a14b63a] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:28:51,604 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-33) [4a14b63a] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:28:51,617 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-33) [4a14b63a] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 379b22d0 >2012-11-08 10:28:51,640 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-33) [4a14b63a] spmStart polling started: taskId = e709a719-030a-4667-9bd5-3b215208dbf7 >2012-11-08 10:28:52,678 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-33) [4a14b63a] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:28:52,678 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-33) [4a14b63a] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:28:52,678 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-33) [4a14b63a] spmStart polling ended: taskId = e709a719-030a-4667-9bd5-3b215208dbf7 task status = finished >2012-11-08 10:28:52,678 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-33) [4a14b63a] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:28:52,709 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-33) [4a14b63a] spmStart polling ended, spm status: Free >2012-11-08 10:28:52,722 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-33) [4a14b63a] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=e709a719-030a-4667-9bd5-3b215208dbf7), log id: 5feafd32 >2012-11-08 10:28:52,746 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-33) [4a14b63a] FINISH, HSMClearTaskVDSCommand, log id: 5feafd32 >2012-11-08 10:28:52,746 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-33) [4a14b63a] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3c950ed0, log id: 379b22d0 >2012-11-08 10:28:52,748 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-33) [5dff98c2] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:28:52,764 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-33) [5dff98c2] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:29:02,827 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-76) [3a3774e9] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:29:02,857 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-76) [3a3774e9] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:29:02,860 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-76) [3a3774e9] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:29:02,878 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-76) [3a3774e9] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 54306fe8 >2012-11-08 10:29:02,908 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-76) [3a3774e9] spmStart polling started: taskId = e00fb150-3e1b-47f5-bc8d-49be751ecb6f >2012-11-08 10:29:03,941 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-76) [3a3774e9] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:29:03,941 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-76) [3a3774e9] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:29:03,941 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-76) [3a3774e9] spmStart polling ended: taskId = e00fb150-3e1b-47f5-bc8d-49be751ecb6f task status = finished >2012-11-08 10:29:03,941 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-76) [3a3774e9] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:29:03,970 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-76) [3a3774e9] spmStart polling ended, spm status: Free >2012-11-08 10:29:03,980 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-76) [3a3774e9] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=e00fb150-3e1b-47f5-bc8d-49be751ecb6f), log id: fecefcd >2012-11-08 10:29:03,997 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-76) [3a3774e9] FINISH, HSMClearTaskVDSCommand, log id: fecefcd >2012-11-08 10:29:03,997 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-76) [3a3774e9] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@c7383e6, log id: 54306fe8 >2012-11-08 10:29:03,999 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-76) [73d8776c] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:29:04,022 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-76) [73d8776c] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:29:14,077 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-11) [589c1b50] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:29:14,101 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-11) [589c1b50] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:29:14,104 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-11) [589c1b50] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:29:14,120 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-11) [589c1b50] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3eaa1436 >2012-11-08 10:29:14,148 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-11) [589c1b50] spmStart polling started: taskId = ef1721dd-ce69-4677-82e5-b4e7114ddb4f >2012-11-08 10:29:15,185 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-11) [589c1b50] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:29:15,185 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-11) [589c1b50] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:29:15,185 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-11) [589c1b50] spmStart polling ended: taskId = ef1721dd-ce69-4677-82e5-b4e7114ddb4f task status = finished >2012-11-08 10:29:15,185 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-11) [589c1b50] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:29:15,217 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-11) [589c1b50] spmStart polling ended, spm status: Free >2012-11-08 10:29:15,231 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-11) [589c1b50] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=ef1721dd-ce69-4677-82e5-b4e7114ddb4f), log id: a67eb90 >2012-11-08 10:29:15,254 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-11) [589c1b50] FINISH, HSMClearTaskVDSCommand, log id: a67eb90 >2012-11-08 10:29:15,254 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-11) [589c1b50] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2c9a4046, log id: 3eaa1436 >2012-11-08 10:29:15,256 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-11) [42f1ca8a] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:29:15,275 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-11) [42f1ca8a] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:29:25,328 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-52) [ce43775] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:29:25,356 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-52) [ce43775] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:29:25,360 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-52) [ce43775] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:29:25,377 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-52) [ce43775] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 9c5c4f8 >2012-11-08 10:29:25,406 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-52) [ce43775] spmStart polling started: taskId = 958d6ae3-b4f3-4876-af75-7549e77839fc >2012-11-08 10:29:26,452 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-52) [ce43775] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:29:26,453 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-52) [ce43775] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:29:26,453 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-52) [ce43775] spmStart polling ended: taskId = 958d6ae3-b4f3-4876-af75-7549e77839fc task status = finished >2012-11-08 10:29:26,453 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-52) [ce43775] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:29:26,481 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-52) [ce43775] spmStart polling ended, spm status: Free >2012-11-08 10:29:26,492 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-52) [ce43775] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=958d6ae3-b4f3-4876-af75-7549e77839fc), log id: 3069e5ef >2012-11-08 10:29:26,511 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-52) [ce43775] FINISH, HSMClearTaskVDSCommand, log id: 3069e5ef >2012-11-08 10:29:26,511 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-52) [ce43775] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7876c6bd, log id: 9c5c4f8 >2012-11-08 10:29:26,512 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-52) [3df1e014] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:29:26,531 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-52) [3df1e014] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:29:36,578 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [496e083b] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:29:36,607 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [496e083b] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:29:36,610 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [496e083b] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:29:36,627 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [496e083b] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 659ae870 >2012-11-08 10:29:36,650 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [496e083b] spmStart polling started: taskId = 3e1f8fca-b25e-452f-ba1a-f1e3718b0cf8 >2012-11-08 10:29:37,681 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-85) [496e083b] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:29:37,681 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-85) [496e083b] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:29:37,681 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [496e083b] spmStart polling ended: taskId = 3e1f8fca-b25e-452f-ba1a-f1e3718b0cf8 task status = finished >2012-11-08 10:29:37,681 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [496e083b] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:29:37,708 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [496e083b] spmStart polling ended, spm status: Free >2012-11-08 10:29:37,721 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-85) [496e083b] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=3e1f8fca-b25e-452f-ba1a-f1e3718b0cf8), log id: 1700b4a7 >2012-11-08 10:29:37,739 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-85) [496e083b] FINISH, HSMClearTaskVDSCommand, log id: 1700b4a7 >2012-11-08 10:29:37,739 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [496e083b] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2f544bbd, log id: 659ae870 >2012-11-08 10:29:37,741 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-85) [47387d91] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:29:37,760 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [47387d91] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:29:47,811 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [6607bf9b] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:29:47,838 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [6607bf9b] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:29:47,841 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [6607bf9b] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:29:47,857 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-84) [6607bf9b] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 218535cd >2012-11-08 10:29:47,887 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-84) [6607bf9b] spmStart polling started: taskId = 782e6da5-c425-4a75-bab8-4c5de15e2c46 >2012-11-08 10:29:48,919 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-84) [6607bf9b] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:29:48,920 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-84) [6607bf9b] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:29:48,920 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-84) [6607bf9b] spmStart polling ended: taskId = 782e6da5-c425-4a75-bab8-4c5de15e2c46 task status = finished >2012-11-08 10:29:48,920 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-84) [6607bf9b] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:29:48,947 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-84) [6607bf9b] spmStart polling ended, spm status: Free >2012-11-08 10:29:48,960 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-84) [6607bf9b] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=782e6da5-c425-4a75-bab8-4c5de15e2c46), log id: 6a69ed3d >2012-11-08 10:29:48,983 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-84) [6607bf9b] FINISH, HSMClearTaskVDSCommand, log id: 6a69ed3d >2012-11-08 10:29:48,983 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-84) [6607bf9b] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@5056765f, log id: 218535cd >2012-11-08 10:29:48,986 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-84) [5c75802d] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:29:49,011 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-84) [5c75802d] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:29:59,070 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) [11e66ede] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:29:59,093 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) [11e66ede] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:29:59,097 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) [11e66ede] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:29:59,115 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-60) [11e66ede] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 20170365 >2012-11-08 10:29:59,145 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-60) [11e66ede] spmStart polling started: taskId = 9a8afce5-eb19-4cdc-bf1c-91fe480f2a78 >2012-11-08 10:30:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-56) [7784c83b] Autorecovering hosts is disabled, skipping >2012-11-08 10:30:00,000 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-56) [7784c83b] Autorecovering storage domains is disabled, skipping >2012-11-08 10:30:00,183 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-60) [11e66ede] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:30:00,183 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-60) [11e66ede] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:30:00,183 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-60) [11e66ede] spmStart polling ended: taskId = 9a8afce5-eb19-4cdc-bf1c-91fe480f2a78 task status = finished >2012-11-08 10:30:00,183 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-60) [11e66ede] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:30:00,210 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-60) [11e66ede] spmStart polling ended, spm status: Free >2012-11-08 10:30:00,221 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-60) [11e66ede] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=9a8afce5-eb19-4cdc-bf1c-91fe480f2a78), log id: 195021ff >2012-11-08 10:30:00,244 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-60) [11e66ede] FINISH, HSMClearTaskVDSCommand, log id: 195021ff >2012-11-08 10:30:00,244 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-60) [11e66ede] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@5be7f954, log id: 20170365 >2012-11-08 10:30:00,245 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-60) [5aa228ef] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:30:00,270 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-60) [5aa228ef] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:30:10,330 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-100) [52c68d9d] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:30:10,357 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-100) [52c68d9d] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:30:10,360 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-100) [52c68d9d] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:30:10,376 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-100) [52c68d9d] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 596540be >2012-11-08 10:30:10,398 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-100) [52c68d9d] spmStart polling started: taskId = 8192ba27-5610-486c-9b6e-b6dacb386d28 >2012-11-08 10:30:11,431 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-100) [52c68d9d] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:30:11,432 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-100) [52c68d9d] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:30:11,432 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-100) [52c68d9d] spmStart polling ended: taskId = 8192ba27-5610-486c-9b6e-b6dacb386d28 task status = finished >2012-11-08 10:30:11,432 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-100) [52c68d9d] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:30:11,459 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-100) [52c68d9d] spmStart polling ended, spm status: Free >2012-11-08 10:30:11,476 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-100) [52c68d9d] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=8192ba27-5610-486c-9b6e-b6dacb386d28), log id: 6d2e2e38 >2012-11-08 10:30:11,492 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-100) [52c68d9d] FINISH, HSMClearTaskVDSCommand, log id: 6d2e2e38 >2012-11-08 10:30:11,492 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-100) [52c68d9d] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@72dc9687, log id: 596540be >2012-11-08 10:30:11,495 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-100) [701eb3fa] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:30:11,517 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-100) [701eb3fa] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:30:18,253 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-10) [6f8dc286] VM b2 68655de5-7bd9-4bdc-8628-3b2069ec312f moved from Up --> Paused >2012-11-08 10:30:20,402 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-11) [42f1ca8a] VM b3 7fa6f92f-755f-44c3-8f35-b4c56410c87b moved from Up --> Paused >2012-11-08 10:30:21,577 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) [6bd73967] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:30:21,605 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) [6bd73967] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:30:21,608 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) [6bd73967] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:30:21,625 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-21) [6bd73967] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5badd579 >2012-11-08 10:30:21,650 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-21) [6bd73967] spmStart polling started: taskId = bd986874-877d-4ad0-bb75-e14f2c38e877 >2012-11-08 10:30:22,691 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-21) [6bd73967] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:30:22,691 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-21) [6bd73967] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:30:22,691 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-21) [6bd73967] spmStart polling ended: taskId = bd986874-877d-4ad0-bb75-e14f2c38e877 task status = finished >2012-11-08 10:30:22,691 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-21) [6bd73967] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:30:22,719 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-21) [6bd73967] spmStart polling ended, spm status: Free >2012-11-08 10:30:22,730 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-21) [6bd73967] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=bd986874-877d-4ad0-bb75-e14f2c38e877), log id: 29e9b3e4 >2012-11-08 10:30:22,752 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-21) [6bd73967] FINISH, HSMClearTaskVDSCommand, log id: 29e9b3e4 >2012-11-08 10:30:22,752 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-21) [6bd73967] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2e21533b, log id: 5badd579 >2012-11-08 10:30:22,754 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-21) [ddf5e26] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:30:22,779 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) [ddf5e26] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:30:32,837 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [77b9520a] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:30:32,865 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [77b9520a] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:30:32,868 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [77b9520a] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:30:32,885 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) [77b9520a] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3fd47f3d >2012-11-08 10:30:32,914 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) [77b9520a] spmStart polling started: taskId = adaf0fe7-87f9-4662-9901-e8d2e49a6e40 >2012-11-08 10:30:33,952 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-61) [77b9520a] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:30:33,952 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-61) [77b9520a] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:30:33,952 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) [77b9520a] spmStart polling ended: taskId = adaf0fe7-87f9-4662-9901-e8d2e49a6e40 task status = finished >2012-11-08 10:30:33,952 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) [77b9520a] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:30:33,984 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) [77b9520a] spmStart polling ended, spm status: Free >2012-11-08 10:30:33,998 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-61) [77b9520a] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=adaf0fe7-87f9-4662-9901-e8d2e49a6e40), log id: 7a173701 >2012-11-08 10:30:34,021 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-61) [77b9520a] FINISH, HSMClearTaskVDSCommand, log id: 7a173701 >2012-11-08 10:30:34,021 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) [77b9520a] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@a1c6146, log id: 3fd47f3d >2012-11-08 10:30:34,023 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-61) [4d363fd9] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:30:34,040 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [4d363fd9] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:30:44,095 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [47387d91] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:30:44,124 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [47387d91] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:30:44,127 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [47387d91] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:30:44,144 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [47387d91] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 467e9eae >2012-11-08 10:30:44,167 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [47387d91] spmStart polling started: taskId = 37613460-d957-4a05-a9aa-32bc385bcad4 >2012-11-08 10:30:45,207 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-85) [47387d91] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:30:45,207 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-85) [47387d91] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:30:45,207 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [47387d91] spmStart polling ended: taskId = 37613460-d957-4a05-a9aa-32bc385bcad4 task status = finished >2012-11-08 10:30:45,207 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [47387d91] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:30:45,236 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [47387d91] spmStart polling ended, spm status: Free >2012-11-08 10:30:45,253 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-85) [47387d91] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=37613460-d957-4a05-a9aa-32bc385bcad4), log id: 7fbf3faf >2012-11-08 10:30:45,271 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-85) [47387d91] FINISH, HSMClearTaskVDSCommand, log id: 7fbf3faf >2012-11-08 10:30:45,271 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-85) [47387d91] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@64ee3bae, log id: 467e9eae >2012-11-08 10:30:45,273 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-85) [54905046] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:30:45,296 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [54905046] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:30:55,353 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [2709f1e8] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:30:55,377 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [2709f1e8] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:30:55,380 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [2709f1e8] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:30:55,395 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [2709f1e8] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 918584b >2012-11-08 10:30:55,420 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [2709f1e8] spmStart polling started: taskId = 16867959-8376-488c-80ed-163d7d593fa5 >2012-11-08 10:30:56,460 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-14) [2709f1e8] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:30:56,460 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-14) [2709f1e8] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:30:56,460 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [2709f1e8] spmStart polling ended: taskId = 16867959-8376-488c-80ed-163d7d593fa5 task status = finished >2012-11-08 10:30:56,460 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [2709f1e8] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:30:56,487 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [2709f1e8] spmStart polling ended, spm status: Free >2012-11-08 10:30:56,505 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-14) [2709f1e8] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=16867959-8376-488c-80ed-163d7d593fa5), log id: 59d80469 >2012-11-08 10:30:56,522 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-14) [2709f1e8] FINISH, HSMClearTaskVDSCommand, log id: 59d80469 >2012-11-08 10:30:56,523 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-14) [2709f1e8] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@5f0b42b6, log id: 918584b >2012-11-08 10:30:56,525 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-14) [5c48eb79] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:30:56,544 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-14) [5c48eb79] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:31:06,594 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [7b34303d] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:31:06,623 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [7b34303d] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:31:06,625 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [7b34303d] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:31:06,643 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [7b34303d] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2f7e42b5 >2012-11-08 10:31:06,673 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [7b34303d] spmStart polling started: taskId = 43165be8-ac49-4b78-812d-7f52f939b67b >2012-11-08 10:31:07,710 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-94) [7b34303d] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:31:07,711 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-94) [7b34303d] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:31:07,711 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [7b34303d] spmStart polling ended: taskId = 43165be8-ac49-4b78-812d-7f52f939b67b task status = finished >2012-11-08 10:31:07,711 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [7b34303d] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:31:07,744 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [7b34303d] spmStart polling ended, spm status: Free >2012-11-08 10:31:07,757 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-94) [7b34303d] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=43165be8-ac49-4b78-812d-7f52f939b67b), log id: 43e05daf >2012-11-08 10:31:07,778 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-94) [7b34303d] FINISH, HSMClearTaskVDSCommand, log id: 43e05daf >2012-11-08 10:31:07,778 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-94) [7b34303d] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3f26aef9, log id: 2f7e42b5 >2012-11-08 10:31:07,780 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-94) [4a97dd3] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:31:07,799 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-94) [4a97dd3] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:31:17,851 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-88) [77e8199e] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:31:17,879 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-88) [77e8199e] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:31:17,882 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-88) [77e8199e] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:31:17,901 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-88) [77e8199e] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 302a241c >2012-11-08 10:31:17,923 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-88) [77e8199e] spmStart polling started: taskId = 585ad9f0-2596-4b6e-b0f0-a3035eb99440 >2012-11-08 10:31:18,955 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-88) [77e8199e] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:31:18,955 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-88) [77e8199e] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:31:18,955 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-88) [77e8199e] spmStart polling ended: taskId = 585ad9f0-2596-4b6e-b0f0-a3035eb99440 task status = finished >2012-11-08 10:31:18,955 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-88) [77e8199e] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:31:18,978 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-88) [77e8199e] spmStart polling ended, spm status: Free >2012-11-08 10:31:18,994 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-88) [77e8199e] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=585ad9f0-2596-4b6e-b0f0-a3035eb99440), log id: 42238d0e >2012-11-08 10:31:19,033 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-88) [77e8199e] FINISH, HSMClearTaskVDSCommand, log id: 42238d0e >2012-11-08 10:31:19,033 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-88) [77e8199e] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@23d419b4, log id: 302a241c >2012-11-08 10:31:19,036 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-88) [39e38acc] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:31:19,056 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-88) [39e38acc] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:31:29,102 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) [ddf5e26] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:31:29,126 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) [ddf5e26] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:31:29,129 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) [ddf5e26] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:31:29,144 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-21) [ddf5e26] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 44e51b5b >2012-11-08 10:31:29,167 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-21) [ddf5e26] spmStart polling started: taskId = 7dd6d905-120f-4457-b9f9-63c2c0e40056 >2012-11-08 10:31:30,196 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-21) [ddf5e26] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:31:30,196 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-21) [ddf5e26] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:31:30,197 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-21) [ddf5e26] spmStart polling ended: taskId = 7dd6d905-120f-4457-b9f9-63c2c0e40056 task status = finished >2012-11-08 10:31:30,197 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-21) [ddf5e26] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:31:30,224 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-21) [ddf5e26] spmStart polling ended, spm status: Free >2012-11-08 10:31:30,236 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-21) [ddf5e26] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=7dd6d905-120f-4457-b9f9-63c2c0e40056), log id: 6df6075c >2012-11-08 10:31:30,255 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-21) [ddf5e26] FINISH, HSMClearTaskVDSCommand, log id: 6df6075c >2012-11-08 10:31:30,255 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-21) [ddf5e26] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@335d616a, log id: 44e51b5b >2012-11-08 10:31:30,257 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-21) [37a6907e] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:31:30,277 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-21) [37a6907e] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:31:40,342 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [4d363fd9] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:31:40,371 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [4d363fd9] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:31:40,374 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [4d363fd9] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:31:40,391 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) [4d363fd9] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3bb2bb9a >2012-11-08 10:31:40,421 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) [4d363fd9] spmStart polling started: taskId = bee3150c-8348-4178-b4b2-4fa3a3fbc181 >2012-11-08 10:31:41,467 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-61) [4d363fd9] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:31:41,467 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-61) [4d363fd9] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:31:41,467 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) [4d363fd9] spmStart polling ended: taskId = bee3150c-8348-4178-b4b2-4fa3a3fbc181 task status = finished >2012-11-08 10:31:41,467 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) [4d363fd9] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:31:41,494 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) [4d363fd9] spmStart polling ended, spm status: Free >2012-11-08 10:31:41,506 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-61) [4d363fd9] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=bee3150c-8348-4178-b4b2-4fa3a3fbc181), log id: 13b7e6db >2012-11-08 10:31:41,528 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-61) [4d363fd9] FINISH, HSMClearTaskVDSCommand, log id: 13b7e6db >2012-11-08 10:31:41,528 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-61) [4d363fd9] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@e953dfa, log id: 3bb2bb9a >2012-11-08 10:31:41,530 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-61) [6817ef90] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:31:41,554 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-61) [6817ef90] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:31:51,620 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-53) [3a904612] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:31:51,649 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-53) [3a904612] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:31:51,654 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-53) [3a904612] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:31:51,671 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-53) [3a904612] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4f42850 >2012-11-08 10:31:51,701 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-53) [3a904612] spmStart polling started: taskId = ecd71d64-a7b4-43fb-881e-73f746ec7900 >2012-11-08 10:31:52,745 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-53) [3a904612] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:31:52,745 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-53) [3a904612] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:31:52,746 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-53) [3a904612] spmStart polling ended: taskId = ecd71d64-a7b4-43fb-881e-73f746ec7900 task status = finished >2012-11-08 10:31:52,746 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-53) [3a904612] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:31:52,774 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-53) [3a904612] spmStart polling ended, spm status: Free >2012-11-08 10:31:52,786 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-53) [3a904612] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=ecd71d64-a7b4-43fb-881e-73f746ec7900), log id: 6149cb60 >2012-11-08 10:31:52,809 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-53) [3a904612] FINISH, HSMClearTaskVDSCommand, log id: 6149cb60 >2012-11-08 10:31:52,809 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-53) [3a904612] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@41f34a89, log id: 4f42850 >2012-11-08 10:31:52,811 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-53) [4ba28a32] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:31:52,835 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-53) [4ba28a32] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:32:02,898 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [1e35ba4d] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:32:02,925 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [1e35ba4d] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:32:02,928 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [1e35ba4d] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:32:02,947 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [1e35ba4d] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 47b76f5d >2012-11-08 10:32:02,974 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [1e35ba4d] spmStart polling started: taskId = b7eef517-34e3-4b47-a379-588c56b7a7e9 >2012-11-08 10:32:04,020 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-25) [1e35ba4d] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:32:04,020 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-25) [1e35ba4d] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:32:04,020 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [1e35ba4d] spmStart polling ended: taskId = b7eef517-34e3-4b47-a379-588c56b7a7e9 task status = finished >2012-11-08 10:32:04,020 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [1e35ba4d] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:32:04,052 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [1e35ba4d] spmStart polling ended, spm status: Free >2012-11-08 10:32:04,064 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-25) [1e35ba4d] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=b7eef517-34e3-4b47-a379-588c56b7a7e9), log id: 192c29ca >2012-11-08 10:32:04,088 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-25) [1e35ba4d] FINISH, HSMClearTaskVDSCommand, log id: 192c29ca >2012-11-08 10:32:04,088 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [1e35ba4d] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@226b99af, log id: 47b76f5d >2012-11-08 10:32:04,089 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-25) [4636c1d0] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:32:04,113 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [4636c1d0] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:32:14,173 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-70) [544c5258] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:32:14,200 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-70) [544c5258] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:32:14,203 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-70) [544c5258] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:32:14,220 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-70) [544c5258] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 412e0a3a >2012-11-08 10:32:14,242 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-70) [544c5258] spmStart polling started: taskId = ef137412-3e5c-4110-9c80-e17145553995 >2012-11-08 10:32:15,273 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-70) [544c5258] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:32:15,273 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-70) [544c5258] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:32:15,273 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-70) [544c5258] spmStart polling ended: taskId = ef137412-3e5c-4110-9c80-e17145553995 task status = finished >2012-11-08 10:32:15,273 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-70) [544c5258] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:32:15,302 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-70) [544c5258] spmStart polling ended, spm status: Free >2012-11-08 10:32:15,314 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-70) [544c5258] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=ef137412-3e5c-4110-9c80-e17145553995), log id: 164741ff >2012-11-08 10:32:15,337 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-70) [544c5258] FINISH, HSMClearTaskVDSCommand, log id: 164741ff >2012-11-08 10:32:15,337 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-70) [544c5258] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@203dd7c6, log id: 412e0a3a >2012-11-08 10:32:15,339 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-70) [3c2118b3] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:32:15,360 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-70) [3c2118b3] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:32:25,055 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [54905046] starting ProcessDomainRecovery for domain e0cce407-ebea-4322-b47e-7aceb73a7a6f:DISREP >2012-11-08 10:32:25,082 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) [54905046] domain e0cce407-ebea-4322-b47e-7aceb73a7a6f:DISREP was reported by all hosts in status UP as problematic. Moving the Domain to NonOperational. >2012-11-08 10:32:25,113 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (QuartzScheduler_Worker-85) [10eccbab] Lock Acquired to object EngineLock [exclusiveLocks= key: e0cce407-ebea-4322-b47e-7aceb73a7a6f value: STORAGE >, sharedLocks= key: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 value: POOL >] >2012-11-08 10:32:25,114 WARN [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (QuartzScheduler_Worker-85) [10eccbab] CanDoAction of action DeactivateStorageDomain failed. Reasons:VAR__TYPE__STORAGE__DOMAIN,VAR__ACTION__DEACTIVATE,ACTION_TYPE_FAILED_STORAGE_DOMAIN_STATUS_ILLEGAL >2012-11-08 10:32:25,114 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (QuartzScheduler_Worker-85) [10eccbab] Lock freed to object EngineLock [exclusiveLocks= key: e0cce407-ebea-4322-b47e-7aceb73a7a6f value: STORAGE >, sharedLocks= key: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 value: POOL >] >2012-11-08 10:32:25,416 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-99) [7f760eef] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:32:25,463 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-99) [7f760eef] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:32:25,486 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-99) [7f760eef] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:32:25,498 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-99) [7f760eef] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7078b2b2 >2012-11-08 10:32:25,521 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-99) [7f760eef] spmStart polling started: taskId = 31cc4a76-e2f1-4b4b-98e0-a200bc69a7e0 >2012-11-08 10:32:26,552 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-99) [7f760eef] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:32:26,552 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-99) [7f760eef] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:32:26,552 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-99) [7f760eef] spmStart polling ended: taskId = 31cc4a76-e2f1-4b4b-98e0-a200bc69a7e0 task status = finished >2012-11-08 10:32:26,552 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-99) [7f760eef] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:32:26,581 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-99) [7f760eef] spmStart polling ended, spm status: Free >2012-11-08 10:32:26,595 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-99) [7f760eef] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=31cc4a76-e2f1-4b4b-98e0-a200bc69a7e0), log id: 7b3b7ca2 >2012-11-08 10:32:26,616 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-99) [7f760eef] FINISH, HSMClearTaskVDSCommand, log id: 7b3b7ca2 >2012-11-08 10:32:26,616 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-99) [7f760eef] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7e73aee2, log id: 7078b2b2 >2012-11-08 10:32:26,617 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-99) [5cdeb849] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:32:26,631 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-99) [5cdeb849] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:32:34,440 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [513981bc] domain e0cce407-ebea-4322-b47e-7aceb73a7a6f:DISREP in problem. vds: rhs-gp-srv11.lab.eng.blr.redhat.com >2012-11-08 10:32:36,677 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-19) [6dfbc2bf] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:32:36,698 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-19) [6dfbc2bf] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:32:36,700 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-19) [6dfbc2bf] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:32:36,710 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-19) [6dfbc2bf] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 25db8fc9 >2012-11-08 10:32:36,733 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-19) [6dfbc2bf] spmStart polling started: taskId = a878a4d0-4fce-46d6-bc7c-4d5ab46d5b38 >2012-11-08 10:32:37,764 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-19) [6dfbc2bf] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:32:37,764 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-19) [6dfbc2bf] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:32:37,764 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-19) [6dfbc2bf] spmStart polling ended: taskId = a878a4d0-4fce-46d6-bc7c-4d5ab46d5b38 task status = finished >2012-11-08 10:32:37,764 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-19) [6dfbc2bf] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:32:37,792 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-19) [6dfbc2bf] spmStart polling ended, spm status: Free >2012-11-08 10:32:37,807 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-19) [6dfbc2bf] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=a878a4d0-4fce-46d6-bc7c-4d5ab46d5b38), log id: 39fe80fc >2012-11-08 10:32:37,832 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-19) [6dfbc2bf] FINISH, HSMClearTaskVDSCommand, log id: 39fe80fc >2012-11-08 10:32:37,832 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-19) [6dfbc2bf] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@73c63e22, log id: 25db8fc9 >2012-11-08 10:32:37,835 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-19) [5c19658f] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:32:37,868 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-19) [5c19658f] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:32:47,925 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-56) [7784c83b] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:32:47,950 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-56) [7784c83b] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:32:47,953 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-56) [7784c83b] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:32:47,969 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-56) [7784c83b] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4e806f74 >2012-11-08 10:32:47,993 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-56) [7784c83b] spmStart polling started: taskId = 59c272fe-5c46-4529-8c0f-8a94eea616fe >2012-11-08 10:32:49,029 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-56) [7784c83b] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:32:49,029 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-56) [7784c83b] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:32:49,029 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-56) [7784c83b] spmStart polling ended: taskId = 59c272fe-5c46-4529-8c0f-8a94eea616fe task status = finished >2012-11-08 10:32:49,029 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-56) [7784c83b] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:32:49,056 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-56) [7784c83b] spmStart polling ended, spm status: Free >2012-11-08 10:32:49,071 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-56) [7784c83b] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=59c272fe-5c46-4529-8c0f-8a94eea616fe), log id: 39a652c7 >2012-11-08 10:32:49,090 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-56) [7784c83b] FINISH, HSMClearTaskVDSCommand, log id: 39a652c7 >2012-11-08 10:32:49,090 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-56) [7784c83b] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@5684a462, log id: 4e806f74 >2012-11-08 10:32:49,092 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-56) [1e444a4e] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:32:49,115 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-56) [1e444a4e] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:32:59,177 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-99) [5cdeb849] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:32:59,206 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-99) [5cdeb849] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:32:59,209 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-99) [5cdeb849] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:32:59,227 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-99) [5cdeb849] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 32c17a9d >2012-11-08 10:32:59,252 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-99) [5cdeb849] spmStart polling started: taskId = 513a6703-6895-4f27-9ff5-d4708ea61373 >2012-11-08 10:33:00,283 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-99) [5cdeb849] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:33:00,283 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-99) [5cdeb849] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:33:00,283 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-99) [5cdeb849] spmStart polling ended: taskId = 513a6703-6895-4f27-9ff5-d4708ea61373 task status = finished >2012-11-08 10:33:00,283 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-99) [5cdeb849] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:33:00,311 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-99) [5cdeb849] spmStart polling ended, spm status: Free >2012-11-08 10:33:00,323 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-99) [5cdeb849] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=513a6703-6895-4f27-9ff5-d4708ea61373), log id: 16d0cb6e >2012-11-08 10:33:00,345 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-99) [5cdeb849] FINISH, HSMClearTaskVDSCommand, log id: 16d0cb6e >2012-11-08 10:33:00,345 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-99) [5cdeb849] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@65dbae75, log id: 32c17a9d >2012-11-08 10:33:00,347 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-99) [399ab791] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:33:00,371 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-99) [399ab791] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:33:10,439 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-26) [1053cf6c] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:33:10,473 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-26) [1053cf6c] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:33:10,476 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-26) [1053cf6c] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:33:10,495 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-26) [1053cf6c] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 119d1844 >2012-11-08 10:33:10,521 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-26) [1053cf6c] spmStart polling started: taskId = b7584f26-43e3-4964-8e42-a3ba913b3d78 >2012-11-08 10:33:11,555 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-26) [1053cf6c] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:33:11,555 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-26) [1053cf6c] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:33:11,555 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-26) [1053cf6c] spmStart polling ended: taskId = b7584f26-43e3-4964-8e42-a3ba913b3d78 task status = finished >2012-11-08 10:33:11,555 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-26) [1053cf6c] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:33:11,582 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-26) [1053cf6c] spmStart polling ended, spm status: Free >2012-11-08 10:33:11,593 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-26) [1053cf6c] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=b7584f26-43e3-4964-8e42-a3ba913b3d78), log id: 49eed5ca >2012-11-08 10:33:11,615 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-26) [1053cf6c] FINISH, HSMClearTaskVDSCommand, log id: 49eed5ca >2012-11-08 10:33:11,615 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-26) [1053cf6c] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@5f15e6b1, log id: 119d1844 >2012-11-08 10:33:11,617 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-26) [16dc9f3c] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:33:11,641 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-26) [16dc9f3c] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:33:21,701 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [21af292c] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:33:21,728 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [21af292c] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:33:21,731 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [21af292c] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:33:21,772 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [21af292c] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 707a5aeb >2012-11-08 10:33:21,818 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [21af292c] spmStart polling started: taskId = abaa466f-ce8a-4a50-b5d2-dd6819d1075e >2012-11-08 10:33:22,871 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-72) [21af292c] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:33:22,871 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-72) [21af292c] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:33:22,871 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [21af292c] spmStart polling ended: taskId = abaa466f-ce8a-4a50-b5d2-dd6819d1075e task status = finished >2012-11-08 10:33:22,871 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [21af292c] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:33:22,899 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [21af292c] spmStart polling ended, spm status: Free >2012-11-08 10:33:22,912 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-72) [21af292c] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=abaa466f-ce8a-4a50-b5d2-dd6819d1075e), log id: b8562c3 >2012-11-08 10:33:22,934 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-72) [21af292c] FINISH, HSMClearTaskVDSCommand, log id: b8562c3 >2012-11-08 10:33:22,934 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-72) [21af292c] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6216f61d, log id: 707a5aeb >2012-11-08 10:33:22,936 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-72) [51781c3a] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:33:22,960 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-72) [51781c3a] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:33:33,020 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [49c06cca] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:33:33,054 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [49c06cca] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:33:33,056 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [49c06cca] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:33:33,075 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-96) [49c06cca] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 72b6bb88 >2012-11-08 10:33:33,106 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-96) [49c06cca] spmStart polling started: taskId = 509e466e-05a4-44a4-8675-9433de6f2c7f >2012-11-08 10:33:34,147 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-96) [49c06cca] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:33:34,147 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-96) [49c06cca] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:33:34,147 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-96) [49c06cca] spmStart polling ended: taskId = 509e466e-05a4-44a4-8675-9433de6f2c7f task status = finished >2012-11-08 10:33:34,147 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-96) [49c06cca] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:33:34,181 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-96) [49c06cca] spmStart polling ended, spm status: Free >2012-11-08 10:33:34,197 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-96) [49c06cca] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=509e466e-05a4-44a4-8675-9433de6f2c7f), log id: 879d7a7 >2012-11-08 10:33:34,215 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-96) [49c06cca] FINISH, HSMClearTaskVDSCommand, log id: 879d7a7 >2012-11-08 10:33:34,215 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-96) [49c06cca] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@773d1949, log id: 72b6bb88 >2012-11-08 10:33:34,217 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-96) [4c4744c7] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:33:34,239 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-96) [4c4744c7] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:33:44,297 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [4636c1d0] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:33:44,325 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [4636c1d0] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:33:44,328 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [4636c1d0] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:33:44,346 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [4636c1d0] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 24770a8c >2012-11-08 10:33:44,375 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [4636c1d0] spmStart polling started: taskId = 51d0d4b1-2b7c-4d80-be7f-d5be468777b9 >2012-11-08 10:33:45,407 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-25) [4636c1d0] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:33:45,407 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-25) [4636c1d0] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:33:45,407 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [4636c1d0] spmStart polling ended: taskId = 51d0d4b1-2b7c-4d80-be7f-d5be468777b9 task status = finished >2012-11-08 10:33:45,407 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [4636c1d0] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:33:45,436 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [4636c1d0] spmStart polling ended, spm status: Free >2012-11-08 10:33:45,455 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-25) [4636c1d0] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=51d0d4b1-2b7c-4d80-be7f-d5be468777b9), log id: 3c064eb7 >2012-11-08 10:33:45,471 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-25) [4636c1d0] FINISH, HSMClearTaskVDSCommand, log id: 3c064eb7 >2012-11-08 10:33:45,471 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-25) [4636c1d0] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4219ef2a, log id: 24770a8c >2012-11-08 10:33:45,473 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-25) [34a4d04d] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:33:45,500 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-25) [34a4d04d] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:33:55,560 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-68) [5db100ee] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:33:55,588 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-68) [5db100ee] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:33:55,590 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-68) [5db100ee] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:33:55,601 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-68) [5db100ee] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4b4de0e0 >2012-11-08 10:33:55,627 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-68) [5db100ee] spmStart polling started: taskId = 8a5502ae-78d2-4e80-aa82-2ef533c4e014 >2012-11-08 10:33:56,665 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-68) [5db100ee] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:33:56,665 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-68) [5db100ee] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:33:56,665 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-68) [5db100ee] spmStart polling ended: taskId = 8a5502ae-78d2-4e80-aa82-2ef533c4e014 task status = finished >2012-11-08 10:33:56,665 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-68) [5db100ee] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:33:56,695 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-68) [5db100ee] spmStart polling ended, spm status: Free >2012-11-08 10:33:56,708 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-68) [5db100ee] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=8a5502ae-78d2-4e80-aa82-2ef533c4e014), log id: 3d895a6c >2012-11-08 10:33:56,732 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-68) [5db100ee] FINISH, HSMClearTaskVDSCommand, log id: 3d895a6c >2012-11-08 10:33:56,732 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-68) [5db100ee] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@48f8804, log id: 4b4de0e0 >2012-11-08 10:33:56,734 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-68) [6b2ac0f2] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:33:56,752 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-68) [6b2ac0f2] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:34:06,818 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [23f4778a] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:34:06,841 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [23f4778a] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:34:06,844 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [23f4778a] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:34:06,854 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [23f4778a] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7df80566 >2012-11-08 10:34:06,884 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [23f4778a] spmStart polling started: taskId = c0d8da5f-485b-4f4b-b4e4-cf503503f619 >2012-11-08 10:34:07,922 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-5) [23f4778a] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:34:07,922 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-5) [23f4778a] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:34:07,922 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [23f4778a] spmStart polling ended: taskId = c0d8da5f-485b-4f4b-b4e4-cf503503f619 task status = finished >2012-11-08 10:34:07,922 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [23f4778a] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:34:07,953 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [23f4778a] spmStart polling ended, spm status: Free >2012-11-08 10:34:07,967 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-5) [23f4778a] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=c0d8da5f-485b-4f4b-b4e4-cf503503f619), log id: 14a7fbf5 >2012-11-08 10:34:07,986 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-5) [23f4778a] FINISH, HSMClearTaskVDSCommand, log id: 14a7fbf5 >2012-11-08 10:34:07,986 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [23f4778a] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@208e9c6b, log id: 7df80566 >2012-11-08 10:34:07,988 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-5) [72bf457f] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:34:08,011 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [72bf457f] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:34:18,069 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-36) [14862cb3] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:34:18,096 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-36) [14862cb3] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:34:18,099 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-36) [14862cb3] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:34:18,115 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-36) [14862cb3] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5c39ef73 >2012-11-08 10:34:18,144 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-36) [14862cb3] spmStart polling started: taskId = 18a612a1-ab51-489f-8147-e8520338c858 >2012-11-08 10:34:19,181 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-36) [14862cb3] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:34:19,182 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-36) [14862cb3] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:34:19,182 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-36) [14862cb3] spmStart polling ended: taskId = 18a612a1-ab51-489f-8147-e8520338c858 task status = finished >2012-11-08 10:34:19,182 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-36) [14862cb3] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:34:19,207 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-36) [14862cb3] spmStart polling ended, spm status: Free >2012-11-08 10:34:19,219 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-36) [14862cb3] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=18a612a1-ab51-489f-8147-e8520338c858), log id: 4fc061e1 >2012-11-08 10:34:19,241 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-36) [14862cb3] FINISH, HSMClearTaskVDSCommand, log id: 4fc061e1 >2012-11-08 10:34:19,241 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-36) [14862cb3] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@5f5a23ae, log id: 5c39ef73 >2012-11-08 10:34:19,243 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-36) [16dca56a] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:34:19,267 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-36) [16dca56a] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:34:29,327 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [291b23a9] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:34:29,354 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [291b23a9] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:34:29,356 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [291b23a9] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:34:29,374 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-50) [291b23a9] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 60156951 >2012-11-08 10:34:29,402 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-50) [291b23a9] spmStart polling started: taskId = cc7ca7a7-4b1b-482a-bc99-a20800600d4c >2012-11-08 10:34:30,435 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-50) [291b23a9] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:34:30,435 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-50) [291b23a9] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:34:30,435 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-50) [291b23a9] spmStart polling ended: taskId = cc7ca7a7-4b1b-482a-bc99-a20800600d4c task status = finished >2012-11-08 10:34:30,435 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-50) [291b23a9] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:34:30,460 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-50) [291b23a9] spmStart polling ended, spm status: Free >2012-11-08 10:34:30,470 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-50) [291b23a9] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=cc7ca7a7-4b1b-482a-bc99-a20800600d4c), log id: 21e2d5ee >2012-11-08 10:34:30,492 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-50) [291b23a9] FINISH, HSMClearTaskVDSCommand, log id: 21e2d5ee >2012-11-08 10:34:30,492 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-50) [291b23a9] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@334b0da3, log id: 60156951 >2012-11-08 10:34:30,494 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-50) [46c62a4] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:34:30,518 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [46c62a4] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:34:40,578 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [7b445077] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:34:40,605 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [7b445077] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:34:40,608 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [7b445077] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:34:40,624 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-17) [7b445077] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3cdfb0eb >2012-11-08 10:34:40,648 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-17) [7b445077] spmStart polling started: taskId = c5767a26-f3c7-4a68-b551-a05c65b4f2de >2012-11-08 10:34:41,689 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-17) [7b445077] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:34:41,689 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-17) [7b445077] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:34:41,689 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-17) [7b445077] spmStart polling ended: taskId = c5767a26-f3c7-4a68-b551-a05c65b4f2de task status = finished >2012-11-08 10:34:41,689 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-17) [7b445077] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:34:41,722 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-17) [7b445077] spmStart polling ended, spm status: Free >2012-11-08 10:34:41,736 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-17) [7b445077] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=c5767a26-f3c7-4a68-b551-a05c65b4f2de), log id: 1f9d7c0c >2012-11-08 10:34:41,759 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-17) [7b445077] FINISH, HSMClearTaskVDSCommand, log id: 1f9d7c0c >2012-11-08 10:34:41,759 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-17) [7b445077] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1c3b080b, log id: 3cdfb0eb >2012-11-08 10:34:41,761 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-17) [678d18a7] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:34:41,778 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [678d18a7] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:34:51,845 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) [28e30178] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:34:51,869 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) [28e30178] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:34:51,871 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) [28e30178] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:34:51,886 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-41) [28e30178] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 1b5adf79 >2012-11-08 10:34:51,909 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-41) [28e30178] spmStart polling started: taskId = f5e37755-6022-4950-a847-1a679214d4bd >2012-11-08 10:34:52,941 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-41) [28e30178] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:34:52,941 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-41) [28e30178] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:34:52,942 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-41) [28e30178] spmStart polling ended: taskId = f5e37755-6022-4950-a847-1a679214d4bd task status = finished >2012-11-08 10:34:52,942 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-41) [28e30178] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:34:52,967 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-41) [28e30178] spmStart polling ended, spm status: Free >2012-11-08 10:34:52,979 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-41) [28e30178] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=f5e37755-6022-4950-a847-1a679214d4bd), log id: 1512f409 >2012-11-08 10:34:53,000 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-41) [28e30178] FINISH, HSMClearTaskVDSCommand, log id: 1512f409 >2012-11-08 10:34:53,000 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-41) [28e30178] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@f5385ae, log id: 1b5adf79 >2012-11-08 10:34:53,006 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-41) [346d6a5e] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:34:53,029 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) [346d6a5e] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:35:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-70) [3c2118b3] Autorecovering hosts is disabled, skipping >2012-11-08 10:35:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-70) [3c2118b3] Autorecovering storage domains is disabled, skipping >2012-11-08 10:35:03,094 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [46c62a4] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:35:03,124 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [46c62a4] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:35:03,127 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [46c62a4] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:35:03,142 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-50) [46c62a4] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 36ff2de7 >2012-11-08 10:35:03,164 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-50) [46c62a4] spmStart polling started: taskId = 51bbc7a5-172d-4b2d-8dc6-20c08c24f021 >2012-11-08 10:35:04,201 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-50) [46c62a4] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:35:04,201 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-50) [46c62a4] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:35:04,201 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-50) [46c62a4] spmStart polling ended: taskId = 51bbc7a5-172d-4b2d-8dc6-20c08c24f021 task status = finished >2012-11-08 10:35:04,201 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-50) [46c62a4] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:35:04,233 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-50) [46c62a4] spmStart polling ended, spm status: Free >2012-11-08 10:35:04,246 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-50) [46c62a4] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=51bbc7a5-172d-4b2d-8dc6-20c08c24f021), log id: 77f1567e >2012-11-08 10:35:04,264 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-50) [46c62a4] FINISH, HSMClearTaskVDSCommand, log id: 77f1567e >2012-11-08 10:35:04,264 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-50) [46c62a4] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@565a7b7, log id: 36ff2de7 >2012-11-08 10:35:04,269 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-50) [54e5e225] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:35:04,286 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-50) [54e5e225] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:35:14,340 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [678d18a7] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:35:14,371 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [678d18a7] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:35:14,373 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [678d18a7] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:35:14,391 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-17) [678d18a7] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7d67fa75 >2012-11-08 10:35:14,413 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-17) [678d18a7] spmStart polling started: taskId = d8249bfe-c82f-43e7-8582-5d78626ed841 >2012-11-08 10:35:15,445 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-17) [678d18a7] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:35:15,445 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-17) [678d18a7] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:35:15,445 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-17) [678d18a7] spmStart polling ended: taskId = d8249bfe-c82f-43e7-8582-5d78626ed841 task status = finished >2012-11-08 10:35:15,445 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-17) [678d18a7] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:35:15,474 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-17) [678d18a7] spmStart polling ended, spm status: Free >2012-11-08 10:35:15,487 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-17) [678d18a7] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=d8249bfe-c82f-43e7-8582-5d78626ed841), log id: 4047acf5 >2012-11-08 10:35:15,505 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-17) [678d18a7] FINISH, HSMClearTaskVDSCommand, log id: 4047acf5 >2012-11-08 10:35:15,505 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-17) [678d18a7] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@21ac29cb, log id: 7d67fa75 >2012-11-08 10:35:15,506 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-17) [2debdfb2] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:35:15,525 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) [2debdfb2] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:35:25,592 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [38ad9113] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:35:25,621 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [38ad9113] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:35:25,624 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [38ad9113] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:35:25,643 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-39) [38ad9113] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7dba148a >2012-11-08 10:35:25,674 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-39) [38ad9113] spmStart polling started: taskId = fc44a428-ae54-441d-9fc4-b9f4e7564c4e >2012-11-08 10:35:26,705 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-39) [38ad9113] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:35:26,705 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-39) [38ad9113] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:35:26,705 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-39) [38ad9113] spmStart polling ended: taskId = fc44a428-ae54-441d-9fc4-b9f4e7564c4e task status = finished >2012-11-08 10:35:26,705 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-39) [38ad9113] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:35:26,734 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-39) [38ad9113] spmStart polling ended, spm status: Free >2012-11-08 10:35:26,746 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-39) [38ad9113] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=fc44a428-ae54-441d-9fc4-b9f4e7564c4e), log id: 24dae3ef >2012-11-08 10:35:26,764 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-39) [38ad9113] FINISH, HSMClearTaskVDSCommand, log id: 24dae3ef >2012-11-08 10:35:26,764 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-39) [38ad9113] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1eaa72d0, log id: 7dba148a >2012-11-08 10:35:26,766 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-39) [4857c56a] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:35:26,784 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-39) [4857c56a] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:35:36,850 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [659bcec8] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:35:36,874 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [659bcec8] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:35:36,877 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [659bcec8] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:35:36,892 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [659bcec8] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: baa7691 >2012-11-08 10:35:36,916 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [659bcec8] spmStart polling started: taskId = 5d3fb433-c2b8-4613-a32a-42a7d02f2329 >2012-11-08 10:35:37,954 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-46) [659bcec8] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:35:37,954 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-46) [659bcec8] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:35:37,954 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [659bcec8] spmStart polling ended: taskId = 5d3fb433-c2b8-4613-a32a-42a7d02f2329 task status = finished >2012-11-08 10:35:37,954 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [659bcec8] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:35:37,983 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [659bcec8] spmStart polling ended, spm status: Free >2012-11-08 10:35:37,995 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-46) [659bcec8] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=5d3fb433-c2b8-4613-a32a-42a7d02f2329), log id: 35b03613 >2012-11-08 10:35:38,018 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-46) [659bcec8] FINISH, HSMClearTaskVDSCommand, log id: 35b03613 >2012-11-08 10:35:38,018 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [659bcec8] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@43ae34ee, log id: baa7691 >2012-11-08 10:35:38,019 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-46) [621517f8] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:35:38,043 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [621517f8] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:35:48,102 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [72bf457f] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:35:48,131 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [72bf457f] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:35:48,134 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [72bf457f] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:35:48,150 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [72bf457f] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 75a7f9c0 >2012-11-08 10:35:48,171 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [72bf457f] spmStart polling started: taskId = 2bd0f356-518c-4fe5-ad4f-bc844bbcf8d3 >2012-11-08 10:35:49,214 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-5) [72bf457f] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:35:49,214 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-5) [72bf457f] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:35:49,214 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [72bf457f] spmStart polling ended: taskId = 2bd0f356-518c-4fe5-ad4f-bc844bbcf8d3 task status = finished >2012-11-08 10:35:49,214 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [72bf457f] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:35:49,248 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [72bf457f] spmStart polling ended, spm status: Free >2012-11-08 10:35:49,269 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-5) [72bf457f] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=2bd0f356-518c-4fe5-ad4f-bc844bbcf8d3), log id: 356d2066 >2012-11-08 10:35:49,288 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-5) [72bf457f] FINISH, HSMClearTaskVDSCommand, log id: 356d2066 >2012-11-08 10:35:49,288 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [72bf457f] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@f6a23f1, log id: 75a7f9c0 >2012-11-08 10:35:49,290 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-5) [15dab1f] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:35:49,314 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [15dab1f] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:35:59,384 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) [346d6a5e] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:35:59,410 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) [346d6a5e] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:35:59,414 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) [346d6a5e] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:35:59,430 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-41) [346d6a5e] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7c6fdae2 >2012-11-08 10:35:59,476 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-41) [346d6a5e] spmStart polling started: taskId = 41940b9a-fbbb-4cb1-9a7f-d8a58a8e37c2 >2012-11-08 10:36:00,516 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-41) [346d6a5e] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:36:00,516 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-41) [346d6a5e] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:36:00,517 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-41) [346d6a5e] spmStart polling ended: taskId = 41940b9a-fbbb-4cb1-9a7f-d8a58a8e37c2 task status = finished >2012-11-08 10:36:00,517 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-41) [346d6a5e] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:36:00,549 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-41) [346d6a5e] spmStart polling ended, spm status: Free >2012-11-08 10:36:00,563 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-41) [346d6a5e] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=41940b9a-fbbb-4cb1-9a7f-d8a58a8e37c2), log id: 526ce381 >2012-11-08 10:36:00,596 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-41) [346d6a5e] FINISH, HSMClearTaskVDSCommand, log id: 526ce381 >2012-11-08 10:36:00,596 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-41) [346d6a5e] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2c62e917, log id: 7c6fdae2 >2012-11-08 10:36:00,598 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-41) [11b4fbc5] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:36:00,613 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) [11b4fbc5] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:36:10,672 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [621517f8] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:36:10,700 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [621517f8] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:36:10,703 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [621517f8] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:36:10,719 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [621517f8] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: f114f5f >2012-11-08 10:36:10,749 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [621517f8] spmStart polling started: taskId = 041cc3b3-f5bc-42c5-8f62-5a330b194aad >2012-11-08 10:36:11,782 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-46) [621517f8] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:36:11,782 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-46) [621517f8] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:36:11,782 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [621517f8] spmStart polling ended: taskId = 041cc3b3-f5bc-42c5-8f62-5a330b194aad task status = finished >2012-11-08 10:36:11,782 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [621517f8] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:36:11,810 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [621517f8] spmStart polling ended, spm status: Free >2012-11-08 10:36:11,823 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-46) [621517f8] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=041cc3b3-f5bc-42c5-8f62-5a330b194aad), log id: 585d7702 >2012-11-08 10:36:11,840 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-46) [621517f8] FINISH, HSMClearTaskVDSCommand, log id: 585d7702 >2012-11-08 10:36:11,840 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [621517f8] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@603051a0, log id: f114f5f >2012-11-08 10:36:11,842 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-46) [43f10b6] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:36:11,860 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [43f10b6] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:36:21,914 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [15dab1f] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:36:21,941 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [15dab1f] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:36:21,944 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [15dab1f] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:36:21,960 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [15dab1f] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 28c9112d >2012-11-08 10:36:21,988 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [15dab1f] spmStart polling started: taskId = ee3cf7b4-05f9-4ec6-bf1a-d5497c9c64c3 >2012-11-08 10:36:23,021 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-5) [15dab1f] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:36:23,021 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-5) [15dab1f] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:36:23,021 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [15dab1f] spmStart polling ended: taskId = ee3cf7b4-05f9-4ec6-bf1a-d5497c9c64c3 task status = finished >2012-11-08 10:36:23,021 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [15dab1f] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:36:23,049 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [15dab1f] spmStart polling ended, spm status: Free >2012-11-08 10:36:23,062 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-5) [15dab1f] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=ee3cf7b4-05f9-4ec6-bf1a-d5497c9c64c3), log id: 3b42fe73 >2012-11-08 10:36:23,085 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-5) [15dab1f] FINISH, HSMClearTaskVDSCommand, log id: 3b42fe73 >2012-11-08 10:36:23,085 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [15dab1f] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2b4f4e6a, log id: 28c9112d >2012-11-08 10:36:23,087 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-5) [63a5275c] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:36:23,104 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [63a5275c] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:36:33,164 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [eca9dba] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:36:33,195 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [eca9dba] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:36:33,198 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [eca9dba] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:36:33,216 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-38) [eca9dba] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6cbb8270 >2012-11-08 10:36:33,238 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-38) [eca9dba] spmStart polling started: taskId = f53e2367-189f-4feb-ba67-d1efd1de1cc1 >2012-11-08 10:36:34,274 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-38) [eca9dba] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:36:34,274 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-38) [eca9dba] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:36:34,274 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-38) [eca9dba] spmStart polling ended: taskId = f53e2367-189f-4feb-ba67-d1efd1de1cc1 task status = finished >2012-11-08 10:36:34,274 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-38) [eca9dba] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:36:34,302 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-38) [eca9dba] spmStart polling ended, spm status: Free >2012-11-08 10:36:34,318 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-38) [eca9dba] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=f53e2367-189f-4feb-ba67-d1efd1de1cc1), log id: 1481fa38 >2012-11-08 10:36:34,336 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-38) [eca9dba] FINISH, HSMClearTaskVDSCommand, log id: 1481fa38 >2012-11-08 10:36:34,336 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-38) [eca9dba] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@61527d0c, log id: 6cbb8270 >2012-11-08 10:36:34,338 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-38) [6e8fd0be] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:36:34,362 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [6e8fd0be] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:36:44,410 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [1c54eabc] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:36:44,438 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [1c54eabc] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:36:44,440 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [1c54eabc] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:36:44,457 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [1c54eabc] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 26e65b38 >2012-11-08 10:36:44,480 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [1c54eabc] spmStart polling started: taskId = 33b00663-2b65-4a5a-a6e2-1c989297611d >2012-11-08 10:36:45,518 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-83) [1c54eabc] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:36:45,518 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-83) [1c54eabc] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:36:45,518 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [1c54eabc] spmStart polling ended: taskId = 33b00663-2b65-4a5a-a6e2-1c989297611d task status = finished >2012-11-08 10:36:45,518 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [1c54eabc] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:36:45,551 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [1c54eabc] spmStart polling ended, spm status: Free >2012-11-08 10:36:45,568 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-83) [1c54eabc] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=33b00663-2b65-4a5a-a6e2-1c989297611d), log id: 7225c721 >2012-11-08 10:36:45,585 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-83) [1c54eabc] FINISH, HSMClearTaskVDSCommand, log id: 7225c721 >2012-11-08 10:36:45,585 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-83) [1c54eabc] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@e348213, log id: 26e65b38 >2012-11-08 10:36:45,587 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-83) [1947b4ae] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:36:45,612 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) [1947b4ae] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:36:55,669 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) [3742c089] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:36:55,698 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) [3742c089] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:36:55,700 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) [3742c089] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:36:55,717 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-2) [3742c089] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6e383edc >2012-11-08 10:36:55,748 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-2) [3742c089] spmStart polling started: taskId = 20ee9c68-bf6b-4bd8-a8cc-62ecb46c2a4b >2012-11-08 10:36:56,785 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-2) [3742c089] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:36:56,785 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-2) [3742c089] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:36:56,785 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-2) [3742c089] spmStart polling ended: taskId = 20ee9c68-bf6b-4bd8-a8cc-62ecb46c2a4b task status = finished >2012-11-08 10:36:56,785 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-2) [3742c089] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:36:56,817 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-2) [3742c089] spmStart polling ended, spm status: Free >2012-11-08 10:36:56,830 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-2) [3742c089] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=20ee9c68-bf6b-4bd8-a8cc-62ecb46c2a4b), log id: 509e4995 >2012-11-08 10:36:56,850 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-2) [3742c089] FINISH, HSMClearTaskVDSCommand, log id: 509e4995 >2012-11-08 10:36:56,850 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-2) [3742c089] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@47f0fe8c, log id: 6e383edc >2012-11-08 10:36:56,851 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-2) [1ed7be3a] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:36:56,872 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) [1ed7be3a] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:37:06,919 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-22) [2bf9ae60] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:37:06,942 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-22) [2bf9ae60] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:37:06,944 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-22) [2bf9ae60] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:37:06,959 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-22) [2bf9ae60] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 36552dfb >2012-11-08 10:37:06,981 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-22) [2bf9ae60] spmStart polling started: taskId = 26c6792a-43a1-4c74-96a0-d90d5b13f5c0 >2012-11-08 10:37:08,011 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-22) [2bf9ae60] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:37:08,011 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-22) [2bf9ae60] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:37:08,011 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-22) [2bf9ae60] spmStart polling ended: taskId = 26c6792a-43a1-4c74-96a0-d90d5b13f5c0 task status = finished >2012-11-08 10:37:08,011 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-22) [2bf9ae60] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:37:08,038 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-22) [2bf9ae60] spmStart polling ended, spm status: Free >2012-11-08 10:37:08,051 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-22) [2bf9ae60] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=26c6792a-43a1-4c74-96a0-d90d5b13f5c0), log id: 4925bf72 >2012-11-08 10:37:08,070 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-22) [2bf9ae60] FINISH, HSMClearTaskVDSCommand, log id: 4925bf72 >2012-11-08 10:37:08,070 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-22) [2bf9ae60] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4d3bfae2, log id: 36552dfb >2012-11-08 10:37:08,071 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-22) [1cd4330b] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:37:08,092 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-22) [1cd4330b] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:37:18,145 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-93) [5789f10f] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:37:18,176 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-93) [5789f10f] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:37:18,179 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-93) [5789f10f] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:37:18,196 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-93) [5789f10f] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 464ab4a0 >2012-11-08 10:37:18,224 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-93) [5789f10f] spmStart polling started: taskId = 5cb9b950-5595-4374-a84c-a5af673a7997 >2012-11-08 10:37:19,263 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-93) [5789f10f] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:37:19,263 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-93) [5789f10f] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:37:19,263 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-93) [5789f10f] spmStart polling ended: taskId = 5cb9b950-5595-4374-a84c-a5af673a7997 task status = finished >2012-11-08 10:37:19,263 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-93) [5789f10f] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:37:19,290 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-93) [5789f10f] spmStart polling ended, spm status: Free >2012-11-08 10:37:19,301 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-93) [5789f10f] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=5cb9b950-5595-4374-a84c-a5af673a7997), log id: 41571522 >2012-11-08 10:37:19,323 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-93) [5789f10f] FINISH, HSMClearTaskVDSCommand, log id: 41571522 >2012-11-08 10:37:19,323 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-93) [5789f10f] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3c2c8cb1, log id: 464ab4a0 >2012-11-08 10:37:19,325 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-93) [10c8d754] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:37:19,350 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-93) [10c8d754] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:37:29,417 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [63a5275c] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:37:29,445 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [63a5275c] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:37:29,448 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [63a5275c] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:37:29,465 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [63a5275c] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 79f14461 >2012-11-08 10:37:29,495 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [63a5275c] spmStart polling started: taskId = e393a0f7-981c-4264-8d08-10546557e0e6 >2012-11-08 10:37:30,528 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-5) [63a5275c] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:37:30,528 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-5) [63a5275c] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:37:30,528 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [63a5275c] spmStart polling ended: taskId = e393a0f7-981c-4264-8d08-10546557e0e6 task status = finished >2012-11-08 10:37:30,528 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [63a5275c] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:37:30,556 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [63a5275c] spmStart polling ended, spm status: Free >2012-11-08 10:37:30,569 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-5) [63a5275c] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=e393a0f7-981c-4264-8d08-10546557e0e6), log id: 2be161dd >2012-11-08 10:37:30,587 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-5) [63a5275c] FINISH, HSMClearTaskVDSCommand, log id: 2be161dd >2012-11-08 10:37:30,587 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [63a5275c] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@59e00aac, log id: 79f14461 >2012-11-08 10:37:30,589 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-5) [3dcd04d] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:37:30,611 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [3dcd04d] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:37:34,441 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [513981bc] starting ProcessDomainRecovery for domain e0cce407-ebea-4322-b47e-7aceb73a7a6f:DISREP >2012-11-08 10:37:34,463 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [513981bc] domain e0cce407-ebea-4322-b47e-7aceb73a7a6f:DISREP was reported by all hosts in status UP as problematic. Moving the Domain to NonOperational. >2012-11-08 10:37:34,479 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (QuartzScheduler_Worker-7) [24ab2e6a] Lock Acquired to object EngineLock [exclusiveLocks= key: e0cce407-ebea-4322-b47e-7aceb73a7a6f value: STORAGE >, sharedLocks= key: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 value: POOL >] >2012-11-08 10:37:34,479 WARN [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (QuartzScheduler_Worker-7) [24ab2e6a] CanDoAction of action DeactivateStorageDomain failed. Reasons:VAR__TYPE__STORAGE__DOMAIN,VAR__ACTION__DEACTIVATE,ACTION_TYPE_FAILED_STORAGE_DOMAIN_STATUS_ILLEGAL >2012-11-08 10:37:34,479 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (QuartzScheduler_Worker-7) [24ab2e6a] Lock freed to object EngineLock [exclusiveLocks= key: e0cce407-ebea-4322-b47e-7aceb73a7a6f value: STORAGE >, sharedLocks= key: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 value: POOL >] >2012-11-08 10:37:40,677 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) [11b4fbc5] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:37:40,705 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) [11b4fbc5] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:37:40,708 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) [11b4fbc5] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:37:40,725 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-41) [11b4fbc5] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 1e20ed16 >2012-11-08 10:37:40,749 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-41) [11b4fbc5] spmStart polling started: taskId = d049e810-c2eb-4523-9f4e-0b257d48a304 >2012-11-08 10:37:41,786 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-41) [11b4fbc5] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:37:41,786 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-41) [11b4fbc5] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:37:41,786 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-41) [11b4fbc5] spmStart polling ended: taskId = d049e810-c2eb-4523-9f4e-0b257d48a304 task status = finished >2012-11-08 10:37:41,786 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-41) [11b4fbc5] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:37:41,814 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-41) [11b4fbc5] spmStart polling ended, spm status: Free >2012-11-08 10:37:41,828 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-41) [11b4fbc5] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=d049e810-c2eb-4523-9f4e-0b257d48a304), log id: 2cf06e6d >2012-11-08 10:37:41,851 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-41) [11b4fbc5] FINISH, HSMClearTaskVDSCommand, log id: 2cf06e6d >2012-11-08 10:37:41,851 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-41) [11b4fbc5] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@5ce1cd27, log id: 1e20ed16 >2012-11-08 10:37:41,853 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-41) [6c63eb14] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:37:41,870 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-41) [6c63eb14] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:37:44,097 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-33) [5dff98c2] domain e0cce407-ebea-4322-b47e-7aceb73a7a6f:DISREP in problem. vds: rhs-gp-srv11.lab.eng.blr.redhat.com >2012-11-08 10:37:51,928 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [43f10b6] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:37:51,957 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [43f10b6] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:37:51,959 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [43f10b6] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:37:51,976 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [43f10b6] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 1ff69a85 >2012-11-08 10:37:51,998 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [43f10b6] spmStart polling started: taskId = c60a8c3c-2592-4ff9-8a42-6152133082a9 >2012-11-08 10:37:53,033 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-46) [43f10b6] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:37:53,033 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-46) [43f10b6] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:37:53,033 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [43f10b6] spmStart polling ended: taskId = c60a8c3c-2592-4ff9-8a42-6152133082a9 task status = finished >2012-11-08 10:37:53,033 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [43f10b6] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:37:53,059 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [43f10b6] spmStart polling ended, spm status: Free >2012-11-08 10:37:53,070 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-46) [43f10b6] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=c60a8c3c-2592-4ff9-8a42-6152133082a9), log id: 21f0f1ea >2012-11-08 10:37:53,087 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-46) [43f10b6] FINISH, HSMClearTaskVDSCommand, log id: 21f0f1ea >2012-11-08 10:37:53,087 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-46) [43f10b6] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@519a4b0e, log id: 1ff69a85 >2012-11-08 10:37:53,090 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-46) [68b5adfc] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:37:53,110 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-46) [68b5adfc] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:38:03,175 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [3dcd04d] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:38:03,199 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [3dcd04d] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:38:03,201 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [3dcd04d] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:38:03,216 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [3dcd04d] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2b251f31 >2012-11-08 10:38:03,239 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [3dcd04d] spmStart polling started: taskId = d056bc79-1ffa-444a-9aca-d0cc7b7f4b61 >2012-11-08 10:38:04,273 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-5) [3dcd04d] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:38:04,273 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-5) [3dcd04d] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:38:04,273 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [3dcd04d] spmStart polling ended: taskId = d056bc79-1ffa-444a-9aca-d0cc7b7f4b61 task status = finished >2012-11-08 10:38:04,273 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [3dcd04d] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:38:04,299 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [3dcd04d] spmStart polling ended, spm status: Free >2012-11-08 10:38:04,310 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-5) [3dcd04d] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=d056bc79-1ffa-444a-9aca-d0cc7b7f4b61), log id: 45e28bfc >2012-11-08 10:38:04,331 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-5) [3dcd04d] FINISH, HSMClearTaskVDSCommand, log id: 45e28bfc >2012-11-08 10:38:04,331 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-5) [3dcd04d] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1c54713c, log id: 2b251f31 >2012-11-08 10:38:04,333 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-5) [6bd628ab] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:38:04,358 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-5) [6bd628ab] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:38:14,412 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [6e8fd0be] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:38:14,442 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [6e8fd0be] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:38:14,445 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [6e8fd0be] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:38:14,463 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-38) [6e8fd0be] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 21e032c9 >2012-11-08 10:38:14,484 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-38) [6e8fd0be] spmStart polling started: taskId = f8ef7e0d-f0b2-456e-80df-94f0989d9466 >2012-11-08 10:38:15,515 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-38) [6e8fd0be] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:38:15,516 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-38) [6e8fd0be] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:38:15,516 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-38) [6e8fd0be] spmStart polling ended: taskId = f8ef7e0d-f0b2-456e-80df-94f0989d9466 task status = finished >2012-11-08 10:38:15,516 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-38) [6e8fd0be] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:38:15,544 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-38) [6e8fd0be] spmStart polling ended, spm status: Free >2012-11-08 10:38:15,562 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-38) [6e8fd0be] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=f8ef7e0d-f0b2-456e-80df-94f0989d9466), log id: 495143fa >2012-11-08 10:38:15,579 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-38) [6e8fd0be] FINISH, HSMClearTaskVDSCommand, log id: 495143fa >2012-11-08 10:38:15,579 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-38) [6e8fd0be] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7645f7b0, log id: 21e032c9 >2012-11-08 10:38:15,581 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-38) [6b888079] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:38:15,621 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [6b888079] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:38:25,684 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) [7fb3f382] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:38:25,712 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) [7fb3f382] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:38:25,715 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) [7fb3f382] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:38:25,732 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-65) [7fb3f382] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2a00af1b >2012-11-08 10:38:25,753 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-65) [7fb3f382] spmStart polling started: taskId = 8188fc9f-bd40-488d-a1de-001f3f264b31 >2012-11-08 10:38:26,790 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-65) [7fb3f382] Failed in HSMGetTaskStatusVDS method >2012-11-08 10:38:26,790 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-65) [7fb3f382] Error code AcquireLockFailure and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:38:26,790 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-65) [7fb3f382] spmStart polling ended: taskId = 8188fc9f-bd40-488d-a1de-001f3f264b31 task status = finished >2012-11-08 10:38:26,791 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-65) [7fb3f382] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock >2012-11-08 10:38:26,821 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-65) [7fb3f382] spmStart polling ended, spm status: Free >2012-11-08 10:38:26,839 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-65) [7fb3f382] START, HSMClearTaskVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, taskId=8188fc9f-bd40-488d-a1de-001f3f264b31), log id: 6228e0c2 >2012-11-08 10:38:26,861 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-65) [7fb3f382] FINISH, HSMClearTaskVDSCommand, log id: 6228e0c2 >2012-11-08 10:38:26,861 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-65) [7fb3f382] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@69bc12e6, log id: 2a00af1b >2012-11-08 10:38:26,863 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-65) [3a7cf045] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6 Type: StoragePool >2012-11-08 10:38:26,885 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-65) [3a7cf045] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2012-11-08 10:38:36,936 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) [1ed7be3a] hostFromVds::selectedVds - rhs-gp-srv11.lab.eng.blr.redhat.com, spmStatus Free, storage pool RHS-REBALANCE >2012-11-08 10:38:36,959 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) [1ed7be3a] SpmStatus on vds 96739be4-2990-11e2-8577-52540035ed48: Free >2012-11-08 10:38:36,961 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-2) [1ed7be3a] starting spm on vds rhs-gp-srv11.lab.eng.blr.redhat.com, storage pool RHS-REBALANCE, prevId 1, LVER 0 >2012-11-08 10:38:36,975 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-2) [1ed7be3a] START, SpmStartVDSCommand(HostName = rhs-gp-srv11.lab.eng.blr.redhat.com, HostId = 96739be4-2990-11e2-8577-52540035ed48, storagePoolId = 3a17dcac-a6e3-4563-a4b0-f27d89b68dd6, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 275929a >2012-11-08 10:38:37,001 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-2) [1ed7be3a] spmStart polling started: taskId = bbe0f1e2-8e4e-4213-8ff5-c7ce20a4ccb1
<b>You cannot view the attachment while viewing its details because your browser does not support IFRAMEs. <a href="attachment.cgi?id=640889">View the attachment on a separate page</a>.</b>
View Attachment As Raw
Actions:
View
Attachments on
bug 874660
:
640887
| 640889 |
641269
|
688169